When an apache worker gets a proxy error, it will not retry talking to
the backend server until the retry timeout expires. We bring up the
proxy server *before* the backend server, and poll it. If we are
running a small number of workers, there is a likely chance that we're
going to hit one that errored before the backend was up, thus failing
for now real reason.
Set this to 0 instead to mean always retry failed connections.
Change-Id: I9e584f087bd375f71ddf0c70f83205c425094a17
Ref: https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypass
tls-proxy is the way we're now doing a standard install using https
between services. There is a lot more work to make services directly
handle https, and having python daemons do that directly is a bit of
an anti pattern. Nothing currently tests this in project-config from
my recent grepping, so in the interest of long term maintenance,
delete it all.
Change-Id: I910df4ceab6f24f3d9c484e0433c93b06f17d6e1
We are facing memory pressure in gate testing. Apache is fairly large so
tune its connection limits down to try and squeeze out more useable
memory. THis should be fine for dev envs, also tlsproxy is not enabled
by default so we can check that this tuning works well on a subset of
jobs before making it default everywhere.
Data comparisons done with gate-tempest-dsvm-neutron-full-ubuntu-xenial
jobs.
Old: http://logs.openstack.org/37/447037/2/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/721fc6f/logs/screen-peakmem_tracker.txt.gz
PID %MEM RSS PPID TIME NLWP WCHAN COMMAND
20504 0.2 16660 19589 00:00:00 34 - /usr/sbin/apache2 -k start
20505 0.2 16600 19589 00:00:00 34 - /usr/sbin/apache2 -k start
20672 0.2 16600 19589 00:00:00 34 - /usr/sbin/apache2 -k start
20503 0.1 14388 19589 00:00:00 34 - /usr/sbin/apache2 -k start
19589 0.1 9964 1 00:00:00 1 - /usr/sbin/apache2 -k start
Total RSS: 74212
New: http://logs.openstack.org/41/446741/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/fa4d2e6/logs/screen-peakmem_tracker.txt.gz
PID %MEM RSS PPID TIME NLWP WCHAN COMMAND
8036 0.1 15316 8018 00:00:01 34 - /usr/sbin/apache2 -k start
8037 0.1 15228 8018 00:00:01 34 - /usr/sbin/apache2 -k start
8018 0.1 8584 1 00:00:00 1 - /usr/sbin/apache2 -k start
Total RSS: 39128
Note RSS here is in KB. Total difference is 35084KB or about
34MB. Not the biggest change, but we seem to be functional and it
almost halves the apache overhead.
Change-Id: If82fa347db140021197a215113df4ce38fb4fd17
This reverts commit e0a37cf21e43fbb4ba3f9f8fa5321a0a0e1bedf1.
This didn't help fixing bug #1630664. Issue seems to be between
client<--->Apache2, not between Apache2<--->eventlet
Change-Id: I092c1bbf0c5848b50fc9e491d1e9211451208a89
Force mod_proxy to immediately close a connection to the backend
after being used, and thus, disable its persistent connection and
pool for that backend.
Let's see if that helps fixing bug #1630664 (the
Connection aborted/ BadStatusLine thing).
We already have an ER query (in queries/1630664.yaml) that should show
whether this is effective.
Change-Id: I03b09f7df5c6e134ec4091a2f8dfe8ef614d1951
We are seeing connection errors to the proxy occasionally. These errors
do not result in a logged http request or error to the backends,
resulting in a theory that the proxy itself may just not be able to
handle the number of connections. More than double the total number of
connections that will be accepted by the proxy in an attempt to fix
this.
Change-Id: Iefa6c43451dd1f95927528d2ce0003c84248847f
Related-bug: 1630664
Currently the x509 certificate setup is done after all the
openstack services have been deployed. This is OK because
none of the services require that the x509 certs exist
when they are being deployed. With the integration of TLS
into the nova novnc proxy (and later spice & serial proxy)
service, x509 certs will need to exist before Nova is
deployed.
The CA setup must thus be moved earlier in the devstack
deployment flow, prior to the setup of any services. One
part of the CA setup, however, fixes up the global cert
bundle locations and this can only be done after the
python requests module is install, thus must remain in
its current location.
Change-Id: Idcd264fb73bb88dc2f4280c53c013dfe4364afff
When tls is enabled, we aren't bringing the logs to the forefront,
which makes it hard to debug when things go wrong. This does that.
Change-Id: I7c6c7e324e16da6b9bfa44f4bad17401ca4ed7e3
This creates log files per proxy vhost and sets the log level to info to
help debug potential issues with tls proxying.
Change-Id: I02a62224662b021b35c293909ba045b4b74e1df8
urllib3 1.18 was released today and contains new more correct hostname
matching that takes into account the ipAddress portion of a certificate
and disallows matching an IP Address against a DNS hostname.
Change-Id: I37d247b68911dc85f55adec6a7952ed321c1b1d8
To support multinode testing where we just copy the CA to all the
instances don't remake the CA if it already exists.
The end result is that you can trusty a single chain and all your
clients will be happy regardless of which host they are talking to.
Change-Id: I90892e6828a59fa37af717361a2f1eed15a87ae4
Stud is now abandonware (see https://github.com/bumptech/stud) and is
not packaged in xenial. Lets use Apache for SSL termination since its
there already.
Change-Id: Ifcba410f5969521e8b3d30f02795541c1661f83a
OS_CACERT was being added directly to the environment rather
than usercc_early. This caused an untrusted CA error to be
thrown.
Ensure that SERVICE_HOST is in the Subject Alt. Names of the
issued TLS server cert. The gate sets it to 127.0.0.1 which
wasn't being handled. Only the FQDN of the host and actual
IP address of the machine were being added.
Change-Id: I8a91dffe1a5263d2bcc99ea406a8556045b52be2
Ia0957b47187c3dcadd46154b17022c4213781112 proposes to have bashate
find instances of setting a local value. The issue is that "local"
always returns 0, thus hiding any failure in the commands running to
set the variable.
This is an automated replacement of such instances
Depends-On: I676c805e8f0401f75cc5367eee83b3d880cdef81
Change-Id: I9c8912a8fd596535589b207d7fc553b9d951d3fe
If the version of python-requests required is higher than
that provided by the operating system, pip will install
it from upstream.
The upstream version provides its own CA certificate bundle
based on the Mozilla bundle, and defaults to that in case
a CA certificate file is not specified for a request.
The distribution-specific packages point to the system-wide
CA bundle that can be managed by tools such as
update-ca-trust (Fedora/RHEL) and update-ca-certificates
(Debian/Ubuntu).
When installing in SSL/TLS mode, either with SSL=True or by
adding tls-proxy to ENABLED_SERVICES, if a non-systemwide
CA bundle is used, then the CA generated by devstack will
not be used causing the installation to fail.
Replace the upstream-provided bundle with a link to the
system bundle when possible.
Change-Id: I651aec93398d583dcdc8323503792df7ca05a7e7
Closes-Bug: #1459789
With gerrit 2.8, and the new change screen, this will trigger syntax
highlighting in gerrit. Thus making reviewing code a lot nicer.
Change-Id: Id238748417ffab53e02d59413dba66f61e724383
Configure nova, cinder, glance, swift and neutron to use SSL
on the endpoints using either SSL natively or via a TLS proxy
using stud.
To enable SSL via proxy, in local.conf add
ENABLED_SERVICES+=,tls-proxy
This will create a new test root CA, a subordinate CA and an SSL
server cert. It uses the value of hostname -f for the certificate
subject. The CA certicates are also added to the system CA bundle.
To enable SSL natively, in local.conf add:
USE_SSL=True
Native SSL by default will also use the devstack-generate root and
subordinate CA.
You can override this on a per-service basis by setting
<SERVICE>_SSL_CERT=/path/to/cert
<SERVICE>_SSL_KEY=/path/to/key
<SERVICE>_SSL_PATH=/path/to/ca
You should also set SERVICE_HOST to the FQDN of the host. This
value defaults to the host IP address.
Change-Id: I36fe56c063ca921131ad98439bd452cb135916ac
Closes-Bug: 1328226
Minor cleanups in a couple ofplaces:
* Config funtions
* Stackforge
* TLS
Combined a couple of smaller changesets that are loosly related
Change-Id: Ifa16f2e4c0eca0ef3401c0dfdc4f3d91809021a5
the tls code was leaking out $service as a global variable, which
was causing all manner of confusing errors in grenade trying to
use that variable name. All lower case vars should be localize.
Change-Id: I74fa597f20ee7c714cab83490b42d874ea93db02
Certificates and the index were left in the data directory after
running unstack. That would break devstack on the next run.
partial blueprint devstack-https
Change-Id: I6eb552a76fb29addf4d02254c027e473d6805df1
Rerunning stack.sh after some failure unrelated to ssl setup will fail
due to certificates already existing in the CA index. Don't regenerate
them instead. This is a workaround making devstack development easier
rather than something typical user would run into.
Change-Id: Icfd4cb5132c8c9297eb73159e592b7006295184f
When using ZSH, the line `if [[ (!$cert && !$key && $ca) ]]` fails
due to a syntax error. Instead of checking the variables as a boolean,
we can simply check if they have a non-zero length. This works in ZSH.
Change-Id: I171ed10a8c0af354e82bd6119508a0c44b6bcd9c
Check that function calls look like ^function foo {$ in bash8, and fix
all existing failures of that check. Add a note to HACKING.rst
Change-Id: Ic19eecb39e0b20273d1bcd551a42fe400d54e938
Allow providing certificates through environment variables to be used
for keystone, and provide the basis for doing this for other services.
It cannot be used in conjunction with tls-proxy as the service provides
it's own encrypted endpoint.
Impletmenting: blueprint devstack-https
Change-Id: I8cf4c9c8c8a6911ae56ebcd14600a9d24cca99a0
Address miscellaneous issues with Markdown formatting in comments which
are consumed by shocco when generating the online documentation.
Change-Id: I953075cdbddbf1f119c6c7e35f039e2e54b79078
for files that don't start with a #! or end in .sh, the added tags
are nice for emacs users to automatically switch to the right mode.
Change-Id: If4b93e106191bc744ccad8420cef20e751cdf902
* Adds lib/tls to create test CA/certs
* Start proxy if 'tls-proxy' is enabled
* Configure keystone service catalog for TLS
* Tear down proxy in unstack.sh
* Set auth protocol and ca-cert chain in openrc
* Add DATA_DIR to stackrc
This is the first in a series of patches to enable TLS support
for the service API endpoints.
Change-Id: Ia1c91dc8f1aaf94fbec9dc71da322559a83d14b6
