66 Commits

Author SHA1 Message Date
Monty Taylor
4b4dcf37b1 Pin python images to 3.7
rehash may have an issue with python 3.8 internals.

Change-Id: I1a9c6150d74f6a646446b65a7456c73dc104fd9e
2019-10-22 03:37:23 +09:00
David Shrewsbury
0ae912cc04 No-op change to rebuild python-builder
Apparently, an update to the python-base image caused a new image
to be uploaded to dockerhub. Because python-builder was not updated,
it is running a slightly older base image which we believe is causing
issues with python wheel packaging due to possible different python
versions.

Change-Id: I38948882131e30c7358a970292621b0280b75aac
2019-10-21 11:22:50 -04:00
Monty Taylor
9ab25e89a9 Several updates because the world is a dark place
A few things have changed and we need to fix them in one go.

Use mirror for installing docker for buildset-registry

While, we need to make this more systemic, that's hanging off of the
mirror rework. For now, since we know all of these jobs are debian
based, just set the mirror location.

Replace use of zuul cloner with git clones

You can never be a prophet in your own hometown. This is now broken
because of the git cache rework, so just replace it.

Update libjemalloc library

python:slim is based on buster now, which has libjemalloc2 not
libjemalloc1.

Remove gerrit repo remote for submodules

A recent change to the base jobs to use prepare-workspace-git
broke the gerrit image builds by actually having the origin
remote by /dev/null as intended. This breaks submodules because
for a few of them where we don't have matching stable branches
the submodule relative path behavior is actually exactly what
we want.

Since we don't care about the remote otherwise, remove the
origin remote before doing the submodule update --init so that
the submodule will clone the refs from the zuul prepared repo.

Change-Id: Ieb5b6bc8711fe971ed3445c7c267306ac4616464
2019-10-19 07:51:29 +09:00
Monty Taylor
5a31c91623 Expose volume for tmp dir, add dirs for hooks and static
Our existing config management puts files into hooks and static.
We can bind mount those in, but having the mount-points in the
image is useful.

The tmp dir is important for gerrit to be able to write
plugin expansions and javaamelody data.

Change-Id: Idd917c268ed7bdead412620dfe3ca842736b7463
2019-10-16 14:57:46 +09:00
Monty Taylor
d2e7da959d Use bazel 1.0.0
1.0.0 has released, which is what gerrit wants now. Use it.

Depends-On: https://review.opendev.org/688555
Change-Id: I6cd76b8cfda3656d6105f9fe96b82a388809375f
2019-10-14 16:25:30 -04:00
Monty Taylor
d9fb5b3faf Install jeepyb into the gerrit images
We need jeepyb installed because the content of the gerrit hook scripts
we install is done via jeepyb commands. Use python-builder so that we
can just install the jeepyb wheel.

Should we maybe transition these hooks into being zuul jobs?

Depends-On: https://review.opendev.org/683146/
Change-Id: I8899885b05d1e9f48b3f354ca22b360b54d455a3
2019-09-19 15:17:14 +02:00
Monty Taylor
940e31f043 Fix gerrit java invocation
We need the right path to java, and we also need to tell it to run
daemon.

Change-Id: I7dfbc5a1c054953b34a02327e6bba50c486110d1
2019-09-19 14:16:30 +02:00
Monty Taylor
072fcca06f Fix files matcher and bazel for gerrit base image
Use latest bazel

It seems 0.27 is now too old. This is what happens when I go on vacation
apparently.

Add in a hack to override the bazelversion. We'll remove this once
https://gerrit-review.googlesource.com/c/gerrit/+/237495 lands and
has been merged up.

Change-Id: Ib7a6d33ce8bf8498fd5cd09b25087dc09acb8df4
2019-09-16 21:20:18 +02:00
Monty Taylor
9bb1c73139 Split out bazel builder and gerrit base image
There is a bunch of duplication which needs to be redone almost never.
Split those into their own images so we can run them once and reuse them.

Change-Id: I923d4bff96dae75eb52a1c271fa52d5ae79933a0
2019-08-26 11:26:23 +02:00
Monty Taylor
3a26ed621a Build gerrit with bazel 0.27.1 instead
0.27 is the first semi-stable release.

Change-Id: I7314c6500841ed13fdc10dc951761f5fde4b28af
2019-08-26 11:26:23 +02:00
Monty Taylor
56ceaf1c40 Remove the extra bazel options
We had some extra bazel options that don't seem to be necessary
anymore now that we are using upstream bazel options appropriately.

Retry the build a couple of times if it goes south, inside of the
build image. This should allow re-use of the cache the second time,
and if there is a temporary error, it should pick up and move
forward.

Change-Id: I5f304acb21fd3a4d40701fc0414ae0c424c838e5
2019-08-26 11:26:19 +02:00
Monty Taylor
80981a52c8 Update bazel flags to be more inline with upstream
During the Gerrit Hackathon, we learned some things about setting
bazel options. Use the ones recommended in upstream docs rather than
these. The outcome should be largely the same.

Change-Id: I32b4c567488f0739fb80f69dc881b9837803575c
2019-08-25 11:34:58 +02:00
Clark Boylan
05e0ffdebc Collect gitea sshd logs
Currently we don't have any logs from our gitea sshd processes because
sshd logs to syslog by default and /dev/log isn't in our containers. You
can ask sshd nicely to log to stderr instead with the -e flag which
docker will pick up and store for us.

Update the sshd command to include -e then use testinfra to check we
collect logs and they are accssible from docker.

Change-Id: Ib7d6d405554c3c30be410bc08c6fee7d4363b096
2019-08-06 13:42:25 -07:00
Monty Taylor
2a46202b9f Build gerrit images for 2.16 and 3.0 as well
Our goal is upgrading to 3.0. To do that we need to upgrade to 2.15, then
to 2.16, then to 3.0. Build all of the images so that we can do that.

2.16 and 3.0 also use bazel, so just use one copy of the Dockerfile for
all three and let zuul check out the repos to the right versions.

Depends-On: https://review.opendev.org/673147
Depends-On: https://review.opendev.org/672320
Change-Id: I35bd278e0c70c871fa44d005c60a987d1d8e3cdc
2019-07-27 11:34:42 -04:00
Monty Taylor
943f66e3e6 Build docker images of gerrit
To provide a stepwise upgrade path from 2.13 running directly to
2.15 in a container, make a container image containing the war we're
using currently. This should let us make a change to how we run the
war without changing the war at all, and then update the war.

Instead of trying to make a clean build for gerrit 2.13 inside of a
builder image, just have it wget the already built wars and jars we
have.

There are pieces of this that duplicate what's being done in puppet,
but in this context it's not immediately clear these are important to
do. However, it's also not clear they're a bad idea.

The gerrit 2.15 build needs a newer bazel. Looking at the CI scripts
that are used by gerrithub, we find that they use bazel 0.26.1
and nodesource v10. Use the bazel image published by google to get
a bazel builder image.

Set gerrit uid/git to 3000 in both images to match the existing
directory ownership so that bindmounting doesn't face permissions
problems.

Change-Id: I3533f01c0859ed50640dcfd98023994c5867c056
2019-07-24 04:40:28 -04:00
Joshua Hesketh
9d51994f41 Remove misplaced html tag
Change-Id: Ifcca3ec33a4486ebcecf941c9817e92af34b84a4
2019-05-30 13:33:45 -07:00
James E. Blair
d74c9fd62a Revert "Add tab to link from repo page to gerrit changes"
This reverts commit fe1b3cee80982fa1ec9c084196dd3b19b3f27f44.

We suspect this may be the cause of some templates going all weird:

* times are showing up as "ago%!(EXTRA string=months%!(EXTRA int64=8))"
* many strings are now showing up as lower cased (eg "explore")

Also, the link to gerrit for nova is "project:openstack/" and is
missing the "nova" portion of the name.

Change-Id: I72a06efd118ad0eae231f5ddf1a9888cb8d35aba
2019-05-29 10:10:17 -07:00
Zuul
3cbd1ac93d Merge "Update opendev website to be more present tense" 2019-05-29 16:12:07 +00:00
Zuul
80a9ac2d9c Merge "Add tab to link from repo page to gerrit changes" 2019-05-29 15:51:38 +00:00
James E. Blair
8ebe74e512 Update haproxy-statsd to python3
Change-Id: I1b5c15f0c47a4bf1261abaf6d4e336aad5339c5b
2019-05-25 06:47:57 -07:00
James E. Blair
5faf89f566 Add haproxy-statsd to haproxy server
Build a container image with the haproxy-statsd script, and run that
along with the haproxy container.

Change-Id: I18be70d339df613bf9a72e115e80a6da876111e0
2019-05-24 15:40:28 -07:00
Joshua Hesketh
fe1b3cee80 Add tab to link from repo page to gerrit changes
This goes to the open changes for the currently viewed repo.

Change-Id: I671d81cbeeb271284e783227de2c66baedfbcb83
2019-05-24 05:22:19 +00:00
Zuul
098425a4a7 Merge "Use --no-cache-dir with pip" 2019-05-23 22:29:00 +00:00
Zuul
406d8fc614 Merge "Clean up apt cache" 2019-05-23 22:09:54 +00:00
Monty Taylor
300f798c29 Update opendev website to be more present tense
Since we're doing the rollout, talking about it in the future tense
is no longer necessary.

Change-Id: Ia9ce9c11d8e38d003095047fa9dff300a147021c
2019-05-23 07:33:23 +00:00
James E. Blair
ead3f63649 Switch to upstream and upgrade gitea to current master (pre-1.9.0)
The current gitea master has our change to avoid indexing extra
refs, so we can start replicating refs/changes and refs/notes to
it.  It also fixes a bug we observed when viewing the index of
the starlingx/integ repo.

This also switches us back to the upstream repo, though since
we're using an intermediate commit, the version displayed in
the web ui will be "6eb53ac570ab9af51fc9cbd79f1db782edce57e0".

The docker entrypoint script has moved, so the Dockerfile is updated
to reflect that.

Change-Id: I47769fc1ca62a39122d96a1fc0c1bfc2caca6a4f
2019-05-22 15:14:01 -07:00
James E. Blair
70b8118ab0 Use local fork of gitea and upgrade to 1.8.0
This has a few emergency local patches while we wait for them to
appear in an upstream release.

This updates the modified templates to match the changes in 1.8.0
upstream.

This also disables the oauth2 service, which is new in 1.8.0.
Without disabling this, gitea tries to generate a JWT secret and
write it to the file, which in our case is read only. If we want
to enable it, we need to add a new JWT_SECRET setting.

Change-Id: I969682bce6ff25b7614ce9265097307ee9cbc6cb
Co-Authored-By: Monty Taylor <mordred@inaugust.com>
2019-05-21 12:16:21 -05:00
Paul Belanger
2c4428aa57 Use --no-cache-dir with pip
This helps to reduce the image size but not creating /root/.cache folder
for pip related files.

Change-Id: I1162d7d6fd2e4c7dd9cde44c964aec610a4dd6d3
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2019-04-28 16:13:07 -04:00
Paul Belanger
736695553e Clean up apt cache
By removing /var/lib/apt/lists it reduces the image size.

Change-Id: Icbe118a2725700b9e5e8da97d062161c5b9a5d2d
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2019-04-28 16:04:18 -04:00
Zuul
4436901673 Merge "Run python with jemalloc in containers" 2019-04-05 23:22:05 +00:00
James E. Blair
5923d9130e Upgrade to gitea 1.7.4
Change-Id: I5a962fedd7ff8362a5117706ddefaf2eaa21e29d
2019-03-15 09:09:42 -07:00
James E. Blair
58476a051a Use OpenDev icon in gitea
This should cause the OpenDev logo to be used in the navbar and all
of the favicon/pwa locations.

Change-Id: Id1acfa8214744381fb8e4e2e9779502dde3565e0
2019-03-15 09:01:29 -07:00
Zuul
4c784ba8d0 Merge "Use opendev logos" 2019-03-11 17:55:18 +00:00
Monty Taylor
d7c0be958d Run python with jemalloc in containers
jemalloc is apparently good with python memory according to

  https://zapier.com/engineering/celery-python-jemalloc/

Also, according to these graphs from Tobias:

  https://paste.pics/581cc286226407ab0be400b94951a7d9

it helps with the apparent memory leak seen in python3.6
and python3.7 with zuul.

There doesn't seem to be any downside to it, so install it
and enable it.

Change-Id: I354bc35e5fa1aee90e1c8b6918c5a70dafd4f990
2019-03-08 15:49:51 -08:00
Monty Taylor
9adc5ce8fe Split python-base into its own Dockerfile
There's no real need to tie these together into a multi-stage
Dockerfile as they don't really share anything. Split them.

Change-Id: Ifd7ccadcd8048eeb57797d60356aec2f9f0d2c80
Depends-On: https://review.openstack.org/641805
2019-03-08 15:49:49 -08:00
James E. Blair
b228fec4c6 Use opendev logos
The stacked opendev logo is added to the home page as an svg.

A 120x120 ping is added as the gitea icon.  It appears in several
places throughout gitea source code, and therefore it's easier to
just replace it using the same filename and format rather than
updating the templates with a new filename.

Change-Id: I888b8913c7b6a1045975f10240d744d09b9355b0
2019-03-08 15:26:15 -08:00
Monty Taylor
48bbcc1dd5 Remove releases link from gitea
We don't want to offer links to tarballs of git exports. For the
overwhelming majority of repos, this is a bogus tarball and people
already have the issue of mistakenly trying to use them on github.

In order to bring these back, we'd want to amend the release process
to either also upload tarballs to gitea (probably once we have gitea
back in k8s) - or maybe be able to create Release objects in gitea
that point to external release storage. We probably also want to
write an upstream patch to allow configuring a repo to never show
an autogenerated git export style release.

Change-Id: Idd02d5816b7d04c9a5dcd55fd1d527379d341b92
Story: 2004627
Task: 29848
2019-03-05 19:51:10 +00:00
Monty Taylor
3b39a5f216 Remove Sign In link from gitea
We don't have any auth system connected to this right now.

Change-Id: I1e4a4ac991ca93f099da5a5a7f352441440acd18
2019-03-01 20:02:26 +00:00
James E. Blair
f74ff2bdb7 Have gitea sshd listen on 222
This will allow us to use host networking since it won't conflict
with existing services.

Change-Id: Ie08d626508cd115561161ac1b60ced538a1c9c67
2019-02-19 11:46:32 -08:00
Zuul
8b2e6e511c Merge "Switch gitea to TLS" 2019-02-14 22:45:45 +00:00
James E. Blair
4ae46fd9c3 Switch gitea to TLS
Gitea supports internal TLS termination as well as redirection of
http to https.  That may be simpler than terminating in a reverse
proxy.  Let's try it.

Also, make the dockerfile match the expectation of the job that
there will be a gitea-init stage.

Also, fix the repository root setting in app.ini.

Change-Id: Ia6b66b53cebdb1afa49fef89a0559615f98052f1
2019-02-14 13:41:40 -08:00
Zuul
939453adac Merge "Update to gitea 1.7.1" 2019-02-14 20:54:30 +00:00
Zuul
643865daa6 Merge "Update to gitea v1.6.3" 2019-02-13 21:39:37 +00:00
Monty Taylor
fe078ed994 Rewrite #nXXX to #LXXX in incoming gitea urls
We can't rewrite hashes with mod_rewrite, as they are client-side
and are not sent to the server. Instead, put in a javascript snippet
to rewrite them in the format that gitea is looking for.

We only need to handle single lines, as cgit doesn't handle ranges.

Change-Id: I6226e420838a88276a86e406c40f6707eb090310
2019-02-07 00:46:57 +00:00
Monty Taylor
ffa3c326c1 Update to gitea 1.7.1
1.7.1 has been released. Let's use it.

Change-Id: I421dd444a5b0adfc3a7deeeff2177b73fef039fc
2019-02-06 09:29:39 -08:00
Monty Taylor
394e8f829f Update to gitea v1.6.3
Change-Id: Id362bc5031e42b19fc6fb4ae0b2890e2f196be66
2019-02-06 09:29:39 -08:00
Monty Taylor
24b0c1b9d2 Add modified repo header template
This removes the activity link, which uselessly shows PR and Issue
activity, as well as the fork, start and watch buttons.

Change-Id: Id40eae211ec9b5165cfc0e38e668ce962cf98d9a
2019-02-05 18:52:21 +00:00
Monty Taylor
b78d2f8ed6
Reorder the copy in the python-builder builder image
It really doesn't actually matter, but if we do the apt-get install
before the COPY, it can increase the caching efficiency by a billionth
of a percent.

Depends-On: https://review.openstack.org/632528
Change-Id: I875812c956af1c4739e0986d267a47652bb5e6f3
2019-01-22 19:14:04 +00:00
Monty Taylor
d3c6bbc7af
Install python requirements directly
There are cases when an issue has been fixed upstream in a
dependency but not yet released and it is desirable to install
that version in a built image. pbr does not support such urls
in requirements files because releasing software that depends
on unreleased software is quite dangerous.

Update assemble and install-from-bindep to install -r requirements.txt
before installing the actual software so that image builds with
such requirements lines will work.

Change-Id: Ie17a66bdc4ea0609ce4e66051f659cbd829fdd5d
2019-01-21 20:50:43 +00:00
Monty Taylor
0e1cd6ee85
Add python-builder docker image
This is a builder image intended to make it easy to build
images for projects that use pbr and bindep.

Also, build a python-base image for use with python-builder

pbrx installed dumb-init in a base image and used it as an
entrypoint for the image. This is a good pattern that we should
continue. Make a python-base image that can be used in the FROM
line of consuming final images.

Change-Id: I8da39e680e9c5ca0d8f603a2645e83fc3be87a8b
2019-01-21 20:06:42 +00:00