Co-Authored-By: James E. Blair <corvus@inaugust.com>
Change-Id: Id8b347483affd710759f9b225bfadb3ce851333c
Depends-On: https://review.openstack.org/596503
Ansible handles rsyslog, so rather than having both systems attempting
to manage the service, just let Ansible do it. However, we still install
the haproxy rsyslog config file in Puppet for now. The downside of that
is that an initial server build, or change to the config file, will not
cause rsyslog to reload. That will need to be done manually until the
rest of this is ported to Ansible.
Change-Id: I19060fddf55e7e9f7030e49592f20a4a7faca429
Now that we've got base server stuff rewritten in ansible, remove the
old puppet versions.
Depends-On: https://review.openstack.org/588326
Change-Id: I5c82fe6fd25b9ddaa77747db377ffa7e8bf23c7b
We haven't been receiving any stats since presumably the server
upgrade, because the statsd package wasn't installed.
Change-Id: I98d2625e5a44e9c416baca2fc60eb5304a08a58a
On a slow computer, this rune fails:
git clone -q --mirror git://git.openstack.org/openstack/nova.git nova </dev/null >log 2>&1
Having straced and tcpdumped, what seems to happen is that the server
sends a list of all the refs and corresponding object ids. Then the
client spends quite a while thinking. Firstly the git client process
sits on the cpu for a while, and then it spends a while statting files
correspondong to the various objects.
This thinking takes much longer than is usually the case for `git
clone' because the nova.git repo contains so many refs (>100K refs).
I was only able to repro the problem on a somewhat overloaded VM, and
only with -q and redirecting output to files. I conjecture that git
does something slightly different if it wants to print progess info.
(The practical problem for me is that this causes our git caching
proxy to be unable to clone nova.git.)
In my tests, the slow system took 2 mins 48 seconds between being told
the last ref, and coming back to start asking for objects. This time
may well be significantly longer on an even slower computer.
I suggest setting the timeout to 15 minutes, which gives a factor of 5
safety margin over what I actually observed.
NB that I have not been able to actually test this change.
Change-Id: I0e6c00564fdc91dcd6933c17965c493553f8aea3
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
This adds a script and associated config/init files to periodically
read stats from HAProxy and send them to statsd/graphite.
Change-Id: I77122feacee406b12b3cd0159449c449f2bd35c1
By default haproxy can only bind to HTTP(S) ports all other ports can't
be bound due to the selinux policy. Simple fix for this is to toggle the
boolean that allows haproxy to bind any port in the selinux policy. Do
this with an exec that first checks if the boolean is set.
Change-Id: I49c8bdc3586fa82cd954a6ef9be27f48f9a623ec
By parameterizing the selinux mode we can set the mode and pass the mode
to other classes so they can make decisions on whether or not to run
selinux commands as part of their configuration.
Depends-On: I21add092d9d09077f2b23760a384f5a5cb91d86a
Change-Id: I86a5bb006b6ab9d64f7fbd894c967428cfaed8f8
Continue the great switch to centos7 for git backends and replace git01
and 02. Since these are the last two backends to be swapped out switch
the balance method back to leastconn as we will go back to having
homogenous git backends and can get away with leastconn balancing.
Change-Id: Ib3180587892bb46b3dc9d6a7dea28a28da85c3b3
We currently see some odd behavior with shallow clones using the
leastconn balance method. Theory is that having a mix of centos6 and
centos7 git backends introduces this as a client may end up talking to
different git versions on the backends. Using a specific backend never
has an issue but using the haproxy load balancer does have problems
occasionally.
Work around this by forcing each client to use a single backend via the
source haproxy balance method. Then when the entire cluster of backends
is centos7 based we can switch back to leastconn balancing.
Change-Id: If78ff254388870f90a8b1f4821f4eef98f616d64
It seems from the logs that we're regularly hitting the max 32
connections limit in haproxy for the git protocol backend under
periods of moderate to high load. Increase this to 256, and also
increase the backlog to 256 so that we can have up to one backlogged
connection per active connection.
In theory 256 should be a sane max since we have leastconn balancing
across 8 backends running persistent git-daemon with its default max
connections value of 32.
Change-Id: I80785348b8abb831a2f833a535a5e4234ce3e7f3
Switch from balancing across backends based on source ip address
hash to least number of connections.
Because we have git repos of many sizes, some of which take seconds
to clone, others minutes, we can end up with a load imbalance across
our git backends. Further, organizations that use NAT to put a
large number of systems behind a single IP address further exacerbate
the imbalance. Switch to least-connections for load balancing to
better utilize our backends, and hopefully smooth peak loads across
them.
Change-Id: I7ea5e2ff0c98c46237a975989bd014149de01b04
Currently we are hitting the limit on cloning nova (and apparently
maybe neutron) where the connection is reset during the
git-upload-pack and the clone fails.
Try increasing the HAProxy defaults.
Change-Id: I0072742f82a54b492395cb7db45e2946c6d841c7
lsof is a very valuable tool for debugging connectivity issues. It is
not installed on our centos6 nodes by default. Install it here so that
we have it available for debugging connections issues.
Change-Id: I836b6bb4e852a2fbbff8d3a1620920de8d03b112
The gate-config-puppet-apply test will try to run everything on a
bare-precise node, so even though this doesn't occur in production it is
being tested.
Change-Id: I13b03a7476e30cbb26dc820bb4cf5b56c4018669
* modules/openstack_project/manifests/git.pp: Give the haproxy stats
socket admin level permissions. Make the socket owned by user root and
group root with mode of 0600. This is necessary to provide instructions
to enable and disable backend servers through the stats socket. Also,
install socat on this server so that commands can be given through the
stats socket.
* modules/openstack_project/manifests/git.pp: Enable git-daemon service
so that it starts running at boot.
Change-Id: Ic38114bdbd9fd1e9ab711e636deaadb15f9c7fe3
The haproxy rsyslog config file was left in the cgit module despite
pulling the haproxy stuff out of cgit and putting it in
openstack_project. Move the rsyslog config file into the
openstack_project module where it belongs.
Change-Id: I0fc45f7497f0f2682bbc5c7ae0da546a71b67188
* manifests/site.pp: Use distinct manifests for git.o.o and git.o.o
backends. This allows for the haproxy server to not serve git content
and purely be a load balancer.
* modules/cgit/manifests/init.pp: Remove haproxy from cgit module.
Remove stale xinetd cleanup. Select git daemon port when selecting
HTTP(S) ports.
* modules/openstack_project/manifests/git.pp: Make git.pp a manifest to
load balance git servers with haproxy.
* modules/openstack_project/manifests/git_backend.pp: New manifest to
manage servers that actually serve git content. They sit behind a load
balancer.
* modules/openstack_project/manifests/review.pp: Stop replicating repos
to git load balancer.
Change-Id: I343a0d1e0a7b93874c2e2299ed974a3304957efb
Like the one on gerrit, but this one adds pack-refs (because
the git server on centos seems to deal with that better).
Change-Id: I1ed497a3690d2c030408010dbd56b6cd916b47fa
* install_modules.sh: Add puppetlabs-haproxy forge module.
* modules/cgit/manifests/init.pp: Add haproxy config to load balance
https, https and git protocol git access. Each git server will host git
http on port 8080, https on port 4443 and git protocol on 29418. These
endpoints will then be load balanced by a single haproxy instance
listening on ports 80, 443 and 9418. The use of haproxy and having
services listen on offset ports to accomodate haproxy is toggleable
using the $balance_git and $behind_proxy boolean flags.
Additionally, configure rsyslog for haproxy.
* modules/cgit/files/rsyslog.haproxy.conf: Enable syslog over UDP on
port 514. This is needed by haproxy to perform logging. Send local0
messages to /var/log/haproxy.log.
* modules/cgit/templates/ssl.conf.erb: Make Apache https listen port
configurable. Remove default virtualhost.
* modules/cgit/templates/httpd.conf.erb: Make Apache http listen port
configurable.
* modules/cgit/templates/git.vhost..erb: Make Apache http(s) listen
ports configuruable. Allow http without redirecting to https as a
fallback option to accomodate CentOS clients.
* modules/openstack_project/manifests/git.pp: Pass load balancer
variables through to the cgit manifest.
* manifests/site.pp: Configure git.o.o to run the load balancer haproxy
and balance across the new gitXX.o.o nodes.
Change-Id: Icefc5923cff9a7c6ce62c1923ec2ea87ebc6474a
Create and define a static directory for cgit to use.
Add initial favicon and logo files and include in cgitrc.
Also include custom CSS and background image.
Change-Id: If1e20d8e62557cc90aafbf92f4291b964400e029
Attempted to put Apache ssl certificates for git.o.o into the
/etc/ssl/ and /etc/ssl/ directories, but this is a bit of a
Debianism. Updating to RedHat style /etc/pki/tls directories.
Change-Id: Icb83ee2601d29ad679e7c8503a406faeb5153727
Of course jeepyb::git doesn't really exist, so class failed in
initial attmept to fix missing variables. Moved variables and
comment to an existing class.
Change-Id: I172588eae4b03427f1ee7862073cd50ca4e5a14e
Includes the jeepyb and required pip modules to the new
git.openstack.org server.
Also installs and sets up the create_cgitrepos.py script for
creating the /etc/cgitrepos configuration file
Change-Id: I06606ca73541dd328f413bfdbd2810fe66ca633b
