Nova is gaining the ability to run TLS over the connection between the
novnc proxy service and the QEMU/KVM compute node VNC server.
This adds a new config param - 'NOVA_CONSOLE_PROXY_COMPUTE_TLS=True' -
which instructs devstack to configure libvirt/QEMU to enable TLS for the
VNC server, and to configure the novncproxy to use TLS when connecting.
NB this use of TLS is distinct from use of TLS for the public facing API
controlled by USE_SSL, they can be enabled independently.
This is done in a generic manner so that it is easy to extend to cover
use of TLS with the SPICE and serial console proxy services too.
Change-Id: Ib29d3f5f18533115b9c51e27b373e92fc0a28d1a
Depends-on: I9cc9a380500715e60bd05aa5c29ee46bc6f8d6c2
Implements bp: websocket-proxy-to-host-security
Various services are returning broken links when running behind
tls-proxy. These issues can be fixed by setting the X-Forwarded-Proto
header in the apache config and letting oslo_middleware parse it.
Change-Id: Ibe5dbdc4644ec812f0435f59319666fc336c195a
Partial-Bug: 1713731
In [1] the definition of the function was changed, adding the service
name as first parameter. Since this seems to have caused failures in
some plugins, at least update the function template accordingly.
[1] Ifcba410f5969521e8b3d30f02795541c1661f83a
Change-Id: I4d03957f8d3a18625f06379fb21aa7ba55e32797
After looking at these for I9881f2e7d51fdd9fc0f7fb3e37179aa53171b531 I
found them not as useful as they could be.
Fix the CustomLog command, that wants the logfile then the format
string (or a nickname, which the LogFormat line wasn't setting). Use
standard micro-second timestamps, and trim the access log to have more
relevant info.
Change-Id: I9f4c8ef38ab9e08aeced7b309d4a5276de07af4b
As part of getting swift's functional testing to work properly through
the tls-proxy we need to increase the allowed request header size in
apache. This was a non issue without tls proxy as requests hit the
eventlet webserver directly which was configured via the swift config
which sets this relatively large limit (by default devstack configures
swift to have a header size limit of 16384).
Now we pass in an optional parameter to start_tls_proxy that includes
the desired header size. lib/swift then passes in the value it also
configures in its swift.conf.
If not explicitly set we default to 8190 which is apache2's default.
Change-Id: Ib2811c8d3cbb49cf94b70294788526b15a798edd
This adds packages to suse for systemd python linkages as well as
apache2 and which. And configures mod_proxy and mod_proxy_uwsgi with
a2enmod.
We also properly query if apache mods are enabled to avoid running
into systemd service restart limits. Enable mod_version across the board
as we use it and it may not be enabled by default (like in SUSE).
Also in addition to enabling mod_ssl we enable the SSL flag so that TLS
will work...
Finally we tell the system to trust the devstack CA.
Change-Id: I3442cebfb2e7c2550733eb95a12fab42e1229ce7
We have to do silly overrides of cert locations for requests for
reasons. If we are running under python3 then we were previously looking
in the wrong location for the requests certs. Update the cert fixing
function to properly use python3 to find the certs if python3 is
enabled.
Change-Id: Id1369da0d812edcf9b1204e9c567f8bfe77c48b2
Switch from sha1 to sha256 and from 1024 bits to 2048 bits. Do this
because things don't like the old inseucre sha1+1024bits combo.
Change-Id: Iae2958969aed0cd880844e19e8055c8bdc7d064d
As described in [1], it seems that mod_wsgi is not "graceful" reload
safe. Upon re-init, it can end up in a segfault loop.
The "reload" (not *restart*) after setting up uwsgi was added with
I1d89be1f1b36f26eaf543b99bde6fdc5701474fe but not causing an issue
until uwsgi was enabled.
We do not notice in the gate, because the TLS setup ends up doing a
restart after this setup. In the period between the
write_uwsgi_config and that restart, Apache is sitting in a segfault
loop, but we never noticed because we don't try talking to it. Other
jobs that don't do any further apache configuration have started
failing, however.
Looking at the original comments around "reload_apache_server" I'm not
sure if it is still necessary. [2] shows it is not used outside these
two calls.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1445540
[2] http://codesearch.openstack.org/?q=reload_apache_server&i=nope&files=&repos=
Closes-Bug: #1686210
Change-Id: I5234bae0595efdcd30305a32bf9c121072a3625e
When an apache worker gets a proxy error, it will not retry talking to
the backend server until the retry timeout expires. We bring up the
proxy server *before* the backend server, and poll it. If we are
running a small number of workers, there is a likely chance that we're
going to hit one that errored before the backend was up, thus failing
for now real reason.
Set this to 0 instead to mean always retry failed connections.
Change-Id: I9e584f087bd375f71ddf0c70f83205c425094a17
Ref: https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypass
tls-proxy is the way we're now doing a standard install using https
between services. There is a lot more work to make services directly
handle https, and having python daemons do that directly is a bit of
an anti pattern. Nothing currently tests this in project-config from
my recent grepping, so in the interest of long term maintenance,
delete it all.
Change-Id: I910df4ceab6f24f3d9c484e0433c93b06f17d6e1
We are facing memory pressure in gate testing. Apache is fairly large so
tune its connection limits down to try and squeeze out more useable
memory. THis should be fine for dev envs, also tlsproxy is not enabled
by default so we can check that this tuning works well on a subset of
jobs before making it default everywhere.
Data comparisons done with gate-tempest-dsvm-neutron-full-ubuntu-xenial
jobs.
Old: http://logs.openstack.org/37/447037/2/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/721fc6f/logs/screen-peakmem_tracker.txt.gz
PID %MEM RSS PPID TIME NLWP WCHAN COMMAND
20504 0.2 16660 19589 00:00:00 34 - /usr/sbin/apache2 -k start
20505 0.2 16600 19589 00:00:00 34 - /usr/sbin/apache2 -k start
20672 0.2 16600 19589 00:00:00 34 - /usr/sbin/apache2 -k start
20503 0.1 14388 19589 00:00:00 34 - /usr/sbin/apache2 -k start
19589 0.1 9964 1 00:00:00 1 - /usr/sbin/apache2 -k start
Total RSS: 74212
New: http://logs.openstack.org/41/446741/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/fa4d2e6/logs/screen-peakmem_tracker.txt.gz
PID %MEM RSS PPID TIME NLWP WCHAN COMMAND
8036 0.1 15316 8018 00:00:01 34 - /usr/sbin/apache2 -k start
8037 0.1 15228 8018 00:00:01 34 - /usr/sbin/apache2 -k start
8018 0.1 8584 1 00:00:00 1 - /usr/sbin/apache2 -k start
Total RSS: 39128
Note RSS here is in KB. Total difference is 35084KB or about
34MB. Not the biggest change, but we seem to be functional and it
almost halves the apache overhead.
Change-Id: If82fa347db140021197a215113df4ce38fb4fd17
This reverts commit e0a37cf21e43fbb4ba3f9f8fa5321a0a0e1bedf1.
This didn't help fixing bug #1630664. Issue seems to be between
client<--->Apache2, not between Apache2<--->eventlet
Change-Id: I092c1bbf0c5848b50fc9e491d1e9211451208a89
Force mod_proxy to immediately close a connection to the backend
after being used, and thus, disable its persistent connection and
pool for that backend.
Let's see if that helps fixing bug #1630664 (the
Connection aborted/ BadStatusLine thing).
We already have an ER query (in queries/1630664.yaml) that should show
whether this is effective.
Change-Id: I03b09f7df5c6e134ec4091a2f8dfe8ef614d1951
We are seeing connection errors to the proxy occasionally. These errors
do not result in a logged http request or error to the backends,
resulting in a theory that the proxy itself may just not be able to
handle the number of connections. More than double the total number of
connections that will be accepted by the proxy in an attempt to fix
this.
Change-Id: Iefa6c43451dd1f95927528d2ce0003c84248847f
Related-bug: 1630664
Currently the x509 certificate setup is done after all the
openstack services have been deployed. This is OK because
none of the services require that the x509 certs exist
when they are being deployed. With the integration of TLS
into the nova novnc proxy (and later spice & serial proxy)
service, x509 certs will need to exist before Nova is
deployed.
The CA setup must thus be moved earlier in the devstack
deployment flow, prior to the setup of any services. One
part of the CA setup, however, fixes up the global cert
bundle locations and this can only be done after the
python requests module is install, thus must remain in
its current location.
Change-Id: Idcd264fb73bb88dc2f4280c53c013dfe4364afff
When tls is enabled, we aren't bringing the logs to the forefront,
which makes it hard to debug when things go wrong. This does that.
Change-Id: I7c6c7e324e16da6b9bfa44f4bad17401ca4ed7e3
This creates log files per proxy vhost and sets the log level to info to
help debug potential issues with tls proxying.
Change-Id: I02a62224662b021b35c293909ba045b4b74e1df8
urllib3 1.18 was released today and contains new more correct hostname
matching that takes into account the ipAddress portion of a certificate
and disallows matching an IP Address against a DNS hostname.
Change-Id: I37d247b68911dc85f55adec6a7952ed321c1b1d8
To support multinode testing where we just copy the CA to all the
instances don't remake the CA if it already exists.
The end result is that you can trusty a single chain and all your
clients will be happy regardless of which host they are talking to.
Change-Id: I90892e6828a59fa37af717361a2f1eed15a87ae4
Stud is now abandonware (see https://github.com/bumptech/stud) and is
not packaged in xenial. Lets use Apache for SSL termination since its
there already.
Change-Id: Ifcba410f5969521e8b3d30f02795541c1661f83a
OS_CACERT was being added directly to the environment rather
than usercc_early. This caused an untrusted CA error to be
thrown.
Ensure that SERVICE_HOST is in the Subject Alt. Names of the
issued TLS server cert. The gate sets it to 127.0.0.1 which
wasn't being handled. Only the FQDN of the host and actual
IP address of the machine were being added.
Change-Id: I8a91dffe1a5263d2bcc99ea406a8556045b52be2
Ia0957b47187c3dcadd46154b17022c4213781112 proposes to have bashate
find instances of setting a local value. The issue is that "local"
always returns 0, thus hiding any failure in the commands running to
set the variable.
This is an automated replacement of such instances
Depends-On: I676c805e8f0401f75cc5367eee83b3d880cdef81
Change-Id: I9c8912a8fd596535589b207d7fc553b9d951d3fe
If the version of python-requests required is higher than
that provided by the operating system, pip will install
it from upstream.
The upstream version provides its own CA certificate bundle
based on the Mozilla bundle, and defaults to that in case
a CA certificate file is not specified for a request.
The distribution-specific packages point to the system-wide
CA bundle that can be managed by tools such as
update-ca-trust (Fedora/RHEL) and update-ca-certificates
(Debian/Ubuntu).
When installing in SSL/TLS mode, either with SSL=True or by
adding tls-proxy to ENABLED_SERVICES, if a non-systemwide
CA bundle is used, then the CA generated by devstack will
not be used causing the installation to fail.
Replace the upstream-provided bundle with a link to the
system bundle when possible.
Change-Id: I651aec93398d583dcdc8323503792df7ca05a7e7
Closes-Bug: #1459789
With gerrit 2.8, and the new change screen, this will trigger syntax
highlighting in gerrit. Thus making reviewing code a lot nicer.
Change-Id: Id238748417ffab53e02d59413dba66f61e724383
Configure nova, cinder, glance, swift and neutron to use SSL
on the endpoints using either SSL natively or via a TLS proxy
using stud.
To enable SSL via proxy, in local.conf add
ENABLED_SERVICES+=,tls-proxy
This will create a new test root CA, a subordinate CA and an SSL
server cert. It uses the value of hostname -f for the certificate
subject. The CA certicates are also added to the system CA bundle.
To enable SSL natively, in local.conf add:
USE_SSL=True
Native SSL by default will also use the devstack-generate root and
subordinate CA.
You can override this on a per-service basis by setting
<SERVICE>_SSL_CERT=/path/to/cert
<SERVICE>_SSL_KEY=/path/to/key
<SERVICE>_SSL_PATH=/path/to/ca
You should also set SERVICE_HOST to the FQDN of the host. This
value defaults to the host IP address.
Change-Id: I36fe56c063ca921131ad98439bd452cb135916ac
Closes-Bug: 1328226
Minor cleanups in a couple ofplaces:
* Config funtions
* Stackforge
* TLS
Combined a couple of smaller changesets that are loosly related
Change-Id: Ifa16f2e4c0eca0ef3401c0dfdc4f3d91809021a5
the tls code was leaking out $service as a global variable, which
was causing all manner of confusing errors in grenade trying to
use that variable name. All lower case vars should be localize.
Change-Id: I74fa597f20ee7c714cab83490b42d874ea93db02
Certificates and the index were left in the data directory after
running unstack. That would break devstack on the next run.
partial blueprint devstack-https
Change-Id: I6eb552a76fb29addf4d02254c027e473d6805df1
Rerunning stack.sh after some failure unrelated to ssl setup will fail
due to certificates already existing in the CA index. Don't regenerate
them instead. This is a workaround making devstack development easier
rather than something typical user would run into.
Change-Id: Icfd4cb5132c8c9297eb73159e592b7006295184f
When using ZSH, the line `if [[ (!$cert && !$key && $ca) ]]` fails
due to a syntax error. Instead of checking the variables as a boolean,
we can simply check if they have a non-zero length. This works in ZSH.
Change-Id: I171ed10a8c0af354e82bd6119508a0c44b6bcd9c
Check that function calls look like ^function foo {$ in bash8, and fix
all existing failures of that check. Add a note to HACKING.rst
Change-Id: Ic19eecb39e0b20273d1bcd551a42fe400d54e938
Allow providing certificates through environment variables to be used
for keystone, and provide the basis for doing this for other services.
It cannot be used in conjunction with tls-proxy as the service provides
it's own encrypted endpoint.
Impletmenting: blueprint devstack-https
Change-Id: I8cf4c9c8c8a6911ae56ebcd14600a9d24cca99a0
Address miscellaneous issues with Markdown formatting in comments which
are consumed by shocco when generating the online documentation.
Change-Id: I953075cdbddbf1f119c6c7e35f039e2e54b79078
for files that don't start with a #! or end in .sh, the added tags
are nice for emacs users to automatically switch to the right mode.
Change-Id: If4b93e106191bc744ccad8420cef20e751cdf902
* Adds lib/tls to create test CA/certs
* Start proxy if 'tls-proxy' is enabled
* Configure keystone service catalog for TLS
* Tear down proxy in unstack.sh
* Set auth protocol and ca-cert chain in openrc
* Add DATA_DIR to stackrc
This is the first in a series of patches to enable TLS support
for the service API endpoints.
Change-Id: Ia1c91dc8f1aaf94fbec9dc71da322559a83d14b6
