I476674036748d284b9f51e30cc2ffc9650a50541 did not open port 3081 so
the proxy isn't visible.  Also this group variable is a better place
to update the setting.

Change-Id: Iad0696221bb9a19852e4ce7cbe06b06ab360cf11

We have decided to go with the layer 7 reject rules; enable the
reverse proxy for production hosts.

Change-Id: I476674036748d284b9f51e30cc2ffc9650a50541

This is just spawning containers that never die.

Change-Id: I1f5215c6e60ac59d1eb224bef9032785938dfc70

This brings in the settings added with
I87c85f82f6d38506977bc9bf26d34f6e66746b01 to the container deployment.
As noted there, this stops statsd writing null values for sparesly
updated timers and counters.

Change-Id: I14b5ee40fc8efddfb7bad4fad8a8ae66746131d9

There is a new release, update base container.  Add promote job that
was forgotten with the original commit
Iddfafe852166fe95b3e433420e2e2a4a6380fc64.

Change-Id: Ie0d7febd2686d267903b29dfeda54e7cd6ad77a3

These are to be deployed with containers

Depends-On: https://review.opendev.org/739162
Change-Id: I595f203db86d74bdc31e09a04dbcac0eb906b080

This deploys graphite from the upstream container.

We override the statsd configuration to have it listen on ipv6.
Similarly we override the ngnix config to listen on ipv6, enable ssl,
forward port 80 to 443, block the /admin page (we don't use it).

For production we will just want to put some cinder storage in
/opt/graphite/storage on the production host and figure out how to
migrate the old stats.  The is also a bit of cleanup that will follow,
because we half-converted grafana01.opendev.org -- so everything can't
be in the same group till that is gone.

Testing has been added to push some stats and ensure they are seen.

Change-Id: Ie843b3d90a72564ef90805f820c8abc61a71017d

This uses the Grafana container created with
Iddfafe852166fe95b3e433420e2e2a4a6380fc64 to run the
grafana.opendev.org service.

We retain the old model of an Apache reverse-proxy; it's well tested
and understood, it's much easier than trying to map all the SSL
termination/renewal/etc. into the Grafana container and we don't have
to convince ourselves the container is safe to be directly web-facing.

Otherwise this is a fairly straight forward deployment of the
container.  As before, it uses the graph configuration kept in
project-config which is loaded in with grafyaml, which is included in
the container.

Once nice advantage is that it makes it quite easy to develop graphs
locally, using the container which can talk to the public graphite
instance.  The documentation has been updated with a reference on how
to do this.

Change-Id: I0cc76d29b6911aecfebc71e5fdfe7cf4fcd071a4

Change-Id: Ie2195c8d0d9b3832a14a8bc7a1d761eb4aa53ee5

LXC3 is usable with CentOS 8, while lxc2 is not available for it anymore
So it's worth adding it to reduce network related issues in CI

Change-Id: I562a7d8000ecda8790da88f08128c35b1ec4a2c9

As described inline, this crawler is causing us problems as it hits
the backends indiscriminately.  Block it via the known UA strings,
which luckily are old so should not cause real client issues.

Change-Id: I0d78a8b625b69f600e00e8b3ea64576e0fdb84d9

This adds an option to have an Apache based reverse proxy on port 3081
forwarding to 3000.  The idea is that we can use some of the Apache
filtering rules to reject certain traffic if/when required.

It is off by default, but tested in the gate.

Change-Id: Ie34772878d9fb239a5f69f2d7b993cc1f2142930

We use the Ctx.Req object's RemoteAddr value as it should include the
IP:port combo according to https://golang.org/pkg/net/http/#Request. The
default template uses Ctx.RemoteAddr which Macaron attempts to parse for
x-forwarded-for values but this has the problem of stripping out any
port info.

The port info is important for us because we are doing layer 4 load
balancing and not http l7 load balancing. That means the ip:port
mappings are necessary to map between haproxy and gitea logs.

Change-Id: Icea0d3d815c9d8dd2afe2b1bae627510c1d76f99

Adding the tcplog option to an haproxy backend definition overrides
the default log format. Remove it so the supplied default (which we
based on the tcplog built-in default with some additions) will be
used instead.

Change-Id: Id302dede950c1c2ab8e74a662cc3cb1186a6593d

This looks like a very sane default robots.txt.  We can modify it as
required.

Change-Id: I8b9d3aa63538388e319f0216535f7a1d977f4885

When forwarding TCP sockets at OSI layer 4 with haproxy, it helps to
know the ephemeral port from which it sources each connection to the
backend. In this way, backend connections can be mapped to actual
client IP addresses by correlating backend service access logs with
haproxy logs.

Add "[%bi]:%bp" between the frontend name and backend name values
for the default log-format documented here:

    https://www.haproxy.com/blog/haproxy-log-customization/

Change-Id: Ic2623d483d98cd686a85d40bc4f2e8577fb9087f

This will write an NCSA style access.log file to the logs volume.
This will let us see user agents, etc, to aid in troubleshooting.

Change-Id: I64457f631861768928038676545067b80ef7a122

The increase in connection volume is not sustainable for the available memory on the backend servers. We'll likely need to scale the cluster before reattempting this.

This reverts commit 79f363164e.

Change-Id: Ibe64f472633a62df659c6183aa96e095dda7fdbc

We've set maxconn to 4k concurrent connections on the front side of our
haproxy load balancer. Currently that seems to be creating a large
backlog of requests. Looking at cacti it appears that we have maybe up
to ~6-8 times this amount of overhead in resources on the gitea
backends. Be a little conservative and bump this value up by 4x and tune
from there.

Change-Id: I56d43b52c23f251cc632315c3b57e45541722970

This was originally left here because it seemed like a good thing
to test, but it's currently causing thigns to hang. Exclude it
until we know why.

Change-Id: Ibc6f001e1235e9f0d856cc350ed8099e52c706e9

We install the packages in the builder images so that things can
build.

Change-Id: I2e83158a9ba8517ff199c6efb6abf306b76b7d7d

python-builder is for building docker images. As such, we almost
certainly don't want to be prompted for debconf questions.

Set DEBIAN_FRONTEND=noninteractive so that things like krb5-user
can be installed.

Change-Id: Ieff0555a57e7dbc1b7c4a059b1d0351d016a0c86

We list gitea repos to determine if we need to create a repo. If the
repo isn't listed by gitea we create it. New gitea paginates these
listings so we were only getting 30 repos listed when we had far more.
This resulted in us trying to create repos which already exist which is
a gitea http 409 error.

Fix this by paging through the listings until we've seen all the
repos. This should give us a complete listing.

To test this we run our manage-projects playbook twice in the
system-config-run-gitea job. The first pass creates all the new
projects. Then the second pass should noop cleanly.

Change-Id: I73b77b9ddaa0106d4dc0a49c4d4b7751a39a16f9
Co-Authored-By: Jeremy Stanley <fungi@yuggoth.org>

The list owners miss their notifications, and a quick check of the
Exim logs on the listserv indicate the constant flood which
precipitated these blackhole aliases has finally subsided.

This reverts commit 159f012d47.

Change-Id: I0b3f0bb8a0f802e98211abc5aff8b04586e42f73

This is a docker image based on the latest upstream Grafana with
grafyaml also installed inside.  It includes a small script to run a
refresh of the dashboards.

Change-Id: Iddfafe852166fe95b3e433420e2e2a4a6380fc64

We've noticed that our static sites will semi-regularly have
problems due to stale SSL certs served by Apache workers which
predate the latest certificate replacement and haven't terminated
(graceful restart only ends the running workers once they have no
remaining connections). Limit the impact of this by recycling
workers automatically after a reasonable (large) number of
connections.

This implementation is shamelessly stolen from that used in
Ic377f48d1a5a3eecbcb183327c9255134c4364ab for our mirror sites.

Change-Id: I2e5c0bdf012184ebbfccb086b967008bf12582ab
Co-Authored-By: Clark Boylan <clark.boylan@gmail.com>