Commit Graph

66 Commits

Author SHA1 Message Date
Jens Rosenboom
4b59fbb857 Revert "put mysql on a memory diet"
The diet seems to be too strict, jobs failing with "out of sort memory". Needs more investigation before resubmitting.

This reverts commit 1e66388c5f.

Change-Id: Ic10effaaf047eb3527082baab889772c5e57fa90
2017-03-15 21:58:48 +00:00
Amrith Kumar
1e66388c5f put mysql on a memory diet
We propose several MySQL configuration parameter changes (with
explanations) to reduce the memory footprint of MySQL. A demonstration
of the improvement is provided in
https://etherpad.openstack.org/p/change-438668.

As Clint provided some of the descriptions that I've used, I have
listed him as a co-author (thanks Clint). Let this serve as a warning
to all that commetors may be enlisted :)

Change-Id: Icb2d6ea91d3d45a68ce99c817a746b10039479cc
Co-Authored-By: Clint 'SpamapS' Byrum <clint@fewbar.com>
2017-03-02 09:07:12 -05:00
Roman Podoliaka
88b84094ec mysql: set default sql_mode to TRADITIONAL
We currently use a more permisive STRICT_ALL_TABLES mode, but that's
not what modern MySQL versions default to (i.e. TRADITIONAL):

https://dev.mysql.com/doc/refman/5.7/en/sql-mode.html#sql-mode-changes

(non-Devstack deployments will most likely use TRADITIONAL as well)

Due to the fact that we default to TRADITIONAL in oslo.db, this
produces annoying warnings on MySQL 5.7 versions we use in the gate:

    Warning: (3090, u"Changing sql mode 'NO_AUTO_CREATE_USER' is
              deprecated. It will be removed in a future release.")

https://git.openstack.org/cgit/openstack/oslo.db/tree/oslo_db/options.py#n49

Unlike STRICT_ALL_TABLES, TRADITIONAL mode includes NO_AUTO_CREATE_USER,
and MySQL emits this warning on switching it on:

https://dev.mysql.com/worklog/task/?id=8326

So we have two options here:

1) make oslo.db default to STRICT_ALL_TABLES
2) make Devstack default to TRADITIONAL

The latter seems to be more appropriate as:

1) it's what modern MySQL versions default to
2) it's what people are actually using, if they do not override the
   oslo.db default
3) it's more strict

Closes-Bug: #1652452

Change-Id: Ie6d823c9f8465ac9f2ce4825929d1a50438fab45
2017-02-07 18:01:25 +02:00
Jens Rosenboom
9abb26d696 Try to set initial mysql root password also on Ubuntu
On Ubuntu nodes, devstack tries to predefine the initial mysql root
password by doing some debconf-set-selections, but these will not take
effect if the corresponding package has been installed earlier. So
just try to set it every time, like we do on other distros.

Change-Id: I2c167051fc5e53dd0ccf82a60ab085cd9cdea28d
2016-12-07 21:12:55 +01:00
Jenkins
ddeaaf2044 Merge "Enable some serivce when on boot" 2016-01-07 01:38:08 +00:00
Zhang Jinnan
4d8c03a377 Enable some serivce when on boot
Solve the devstack ./rejoin-stack.sh when is reboot-safe in RHEL 7.
Enable mysql, postgresql, rabbitmq-server, openvswitch service when on boot.

Change-Id: I3ce9fc58ccc76092ad08314de1c3c9339ebfb3b5
Related-Bug: #1486833
2016-01-06 16:40:11 +00:00
Ian Wienand
523f488036 Namespace XTRACE commands
I noticed this when debugging some grenade issues failures.

An include of grenade/functions stores the current value of XTRACE
(on) and disables xtrace for the rest of the import.

We then include devstack's "functions" library, which now overwrites
the stored value of XTRACE the current state; i.e. disabled.

When it finishes it restores the prior state (disabled), and then
grenade restores the same value of XTRACE (disabled).

The result is that xtrace is incorrectly disabled until the next time
it just happens to be turned on.

The solution is to name-space the store of the current-value of xtrace
so when we finish sourcing a file, we always restore the tracing value
to what it was when we entered.

Some files had already discovered this.  In general there is
inconsistency around the setting of the variable, and a lot of obvious
copy-paste.  This brings consistency across all files by using
_XTRACE_* prefixes for the sotre/restore of tracing values.

Change-Id: Iba7739eada5711d9c269cb4127fa712e9f961695
2015-11-27 15:36:04 +11:00
Johan Pas
199d857442 Remove brackets from IPv6 address in mysql cfgfile
stack.sh creates a user-specific configuration file ~/.my.cnf for mysql.
If devstack is installed with SERVICE_IP_VERSION=6 option in local.conf,
the IPv6 host address was stored in the ~/.my.cnf file with square
brackets. However mysql does not use bracketing for IPv6 addresses,
resulting in 'Unknown MySQL server host' error when 'mysql' command is
run. With this patch IPv6 host address is written to ~/.my.cnf without
brackets.

Closes-Bug: #1516776
Change-Id: I27a7be8c75cf6b09b4a75dc4c9d09cd36bc5ac81
2015-11-17 01:02:54 +01:00
Julien Danjou
0eec4f86c1 database: fix PostgreSQL connection string
If all databases drivers are loaded, MySQL SQLAlchemy driver
overrides all the other one that might not have set one.
This patches fixes that.

Change-Id: If6d8d08e5b7b7c48ca012677b536d71058def6fd
Closes-Bug: #1493304
2015-09-08 16:15:33 +00:00
Ian Wienand
9c0b9f3024 Use sudo iniset to modify /etc files
The existing mysql code is wrong and not detected as failing [1], and
boto config requires work-arounds [2,3] that are all fairly ugly.  Use
-sudo argument to iniset to handle this.

[1] I24388b5de777995f92d73076524122cf599d6371
[2] I5f4c43bbbe477c570936e2e40ac05cc38febbb3f
[3] Ib7556dac9aaaf2f3c96237e0ca28ed6ae1b1b7ac

Change-Id: Iaceb8d42ce37be728adae6fd0a30a1f9d33d4029
2015-08-31 10:56:32 +10:00
Marian Horban
ea21eb4f69 Remove non-ASCII characters
Change-Id: If1c68e5aab6990617519150d8aeb3f073df2ad17
2015-08-18 19:53:39 +00:00
Tony Breeds
0294ddc735 mysql: Fix mysql config
devstack attempts to set bind-address, sql_mode, default-storage-engine,
max_connections, query_cache_type and query_cache_size.

However the bash command is missing some '&&'s and was omiting
max_connections, query_cache_type and query_cache_size.

Change-Id: I24388b5de777995f92d73076524122cf599d6371
2015-07-21 14:18:38 -05:00
Brian Haley
180f5eb652 Add IPv6 support to devstack infrastructure
By default, most Openstack services are bound to 0.0.0.0
and service endpoints are registered as IPv4 addresses.
With this change we introduce two new variables to control
this behavior:

SERVICE_IP_VERSION - can either be "4" or "6".

When set to "4" (default if not set) devstack will operate
as today - most services will open listen sockets on 0.0.0.0
and service endpoints will be registered using HOST_IP as the
address.

When set to "6" devstack services will open listen sockets on ::
and service endpoints will be registered using HOST_IPV6 as the
address.

There is no support for "4+6", more work is required for that.

HOST_IPV6 - if SERVICE_IP_VERSION=6 this must be an IPv6
address configured on the system.

Some existing services, like the Openvswitch agent, will continue
to use IPv4 addresses for things like tunnel endpoints.  This is
a current restriction in the code and can be updated at a later
time.  This change is just a first step to supporting IPv6-only
control and data planes in devstack.

This change is also partly based on two previous patches,
https://review.openstack.org/#/c/140519/ and
https://review.openstack.org/#/c/176898/

Change-Id: I5c0b775490ce54ab104fd5e89b20fb700212ae74
Co-Authored-By: Sean Collins <sean@coreitpro.com>
Co-Authored-By: Baodong Li <baoli@cisco.com>
Co-Authored-By: Sridhar Gaddam <sridhar.gaddam@enovance.com>
Co-Authored-By: Adam Kacmarsky <adam.kacmarsky@hp.com>
Co-Authored-By: Jeremy Alvis <jeremy.alvis@hp.com>
2015-07-02 15:16:34 -06:00
Clint Byrum
d16bfa48ee Tune mysql a bit better for concurrent operations
With PyMySQL in the projects we can expect things to happen more
concurrently now. The query cache is a hinderance to concurrency, and
more connections will be required.

Change-Id: Icfb8cdbb9ed39cfd7732ad05fe740e01c767af7b
2015-06-18 13:22:35 -07:00
Jenkins
8cd5576a17 Merge "Revert "Revert "change the default to PyMYSQL""" 2015-06-16 11:31:55 +00:00
Jenkins
aadcde06a2 Merge "Install PyMySQL if used" 2015-06-12 18:36:50 +00:00
armando-migliaccio
b3d8822ec8 Revert "Revert "change the default to PyMYSQL""
Some projects (Neutron) seem to be affected more than others, so we should revert this
to allow for a more selective choice of the DB driver on a per project basis.

We can re-enable the use MySQL-python just for Neutron.

This reverts commit de8d29ed8c.

Related-Bug: #1464612

Change-Id: I889f4f8b116c413b300ab9eecc7b428a9a4afb1a
2015-06-12 16:42:32 +00:00
Sean Dague
de8d29ed8c Revert "change the default to PyMYSQL"
The failure rate with neutron is too high to keep this
as the default.

Related-Bug: #1464612

This reverts commit b3798af474.

Change-Id: Ie9550aeb25d472a38e3d3ef6f3711622c9221c46
2015-06-12 10:43:28 +00:00
Julien Danjou
0f63eb3a37 Install PyMySQL if used
Change Ic609ce136061b753ca692b37509a0b29c60bb8b5 switched to PyMySQL by
default but does not make sure it is installed. This is causing gate
failure in Gnocchi for example:

  http://logs.openstack.org/25/186025/3/check/gate-gnocchi-dsvm-functional-file-mysql/eebd773/logs/devstacklog.txt.gz

Change-Id: I23d313220607fcc8acb95ab43f55b7d9899b9b1f
2015-06-12 09:05:12 +02:00
Sean Dague
b3798af474 change the default to PyMYSQL
As discussed in the Liberty Design Summit "Moving apps to Python 3"
cross-project workshop, the way forward in the near future is to
switch to the pure-python PyMySQL library as a default.

    https://etherpad.openstack.org/p/liberty-cross-project-python3

Change-Id: Ic609ce136061b753ca692b37509a0b29c60bb8b5
2015-05-26 15:39:28 +00:00
Sean Dague
37421991b4 optional pymysql support
This allows you to specify MYSQL_DRIVER=PyMySQL and get it in the
environment.

Change-Id: Ic9d75266640b7aa6d7efb6e882d3027e81414059
2015-05-20 14:58:50 -07:00
Sean Dague
60996b1b60 introduce pip_install_gr
This creates a new pip_install_gr that installs from global
requirements allowed versions. Now that stable branches are getting
capped all of devstack needs to be fixed to do things like this.

Change-Id: I8fd0ef2bfc544ca2576fab09d3018f760b8848fe
2015-04-08 10:28:17 -04:00
Jenkins
c392fd3b5f Merge "Add global venv enable/disable knob" 2015-03-26 20:32:31 +00:00
Wiekus Beukes
ec47bc1d72 Add support for Oracle Linux 7 and later.
Most of the changes revolves around using MySQL rather than MariaDB,
plus enabling the addon repos on public-yum.oracle.com.
The patch just touch the areas where there is a divergence between the
Fedora and Oracle distributions and in all other cases the is_fedora
will result in the correct decision to be made and left as is.

Collapsed the is_suse and is_oraclelinux into a single check in
configure_database_mysql and cleanup_database_mysql

Added Oracle Linux to MAINTAINERS.rst

Rather than duplicating most of the Redhat version check code, added
a check in the block to do the determination if it is Oracle Linux

Change-Id: I5f1f15106329eec67aa008b17847fa44863f243f
2015-03-22 21:28:35 -07:00
Dean Troyer
5686dbc45d Add global venv enable/disable knob
Adds USE_VENV to globally enable/disable use of virtual environments.

ADDITIONAL_VENV_PACKAGES is used to manually add packages that do not
appear in requirements.txt or test-requirements.txt to be installed
into each venv.  Database Python bindings are handled this way when
a dataabse service is enabled.

Change-Id: I9cf298b936fd10c95e2ce5f51aab0d49d4b7f37f
2015-03-20 09:48:43 -05:00
Dean Troyer
b1d8e8e274 Build wheel cache for venvs
Building a bunch of virtual envs later is going to be tedious if we do not
pre-cache certain annoying-to-build packages.

* tools/build_wheels.sh: pre-build some wheels for annoying package installs
* list distro package dependencies in files/*/venv
* list packages to pre-build as wheels in files/venv-requirements.txt
* install database Python modules when setting up the database

Change-Id: Idff1ea69a5ca12ba56098e664dbf6924fe6a2e47
2015-02-18 15:13:00 -06:00
Sean Dague
9a413abcd4 add gating up/down script for devstack
This adds the test infrastructure for testing that unstack.sh and
clean.sh do the right thing, and actually stop what's expected. This
is designed to be used in upstream testing to make unstack and clean a
bit more certain.

It includes numerous fixes to make these pass in an errexit
environment with the gate config. The scripts still don't run under
errexit because we don't assume we've handled all possible cleanup safely.

Change-Id: I774dfb2cc934367eef2bb7ea5123197f6da7565b
2015-02-11 06:10:38 -05:00
Attila Fazekas
1f316beb20 Remove rhel6 and py26 support
el6 is shipped with Python 2.6.x which is not expected
to be supported with the openstack kilo release.

For el6 support we need to do lot of thing differently,
which makes the code more complicated.

This change removes el6 and py26 support from devstack.

This change also removed a discontinued (1 year ago)
openSUSE 12.2 code path, which used a similar codepath as el6.

Several comment related to el6 also removed or modified.

Change-Id: Iea0b0c98a5e11fd85bb5e93c099f740fe05d2f3a
2015-01-27 09:22:52 +01:00
Jenkins
955f80a7ef Merge "Enforce UTF-8 encoding when creating databases" 2015-01-20 15:44:07 +00:00
Bob Ball
9033343733 Set password irrespective of mysql server version
Ubuntu recently upgraded from 5.1 to 5.5 so the previous debconf
settings no longer work.  Removing the version number should make
the settings work for all versions of mysql

Change-Id: I6b399a06232364d3ba3bf74430b663e0b8b922ed
2015-01-19 13:20:44 +00:00
Jenkins
62002dd88b Merge "clean mysql better" 2015-01-17 22:47:58 +00:00
Sean Dague
537532931d Make changes such that -o nounset runs
This makes a bunch of variable cleanups that will let -o nounset
function, for the time being we hide nounset behind another setting
variable so that it's not on by default.

Because this is bash, and things are only executed on demand, this
probably only works in the config it was run in. Expect cleaning up
all the paths to be something that takes quite a while.

This also includes a new set of unit tests around the trueorfalse
function, because my change in how it worked, didn't. Tests are good
m'kay.

Change-Id: I71a896623ea9e1f042a73dc0678ce85acf0dc87d
2015-01-15 13:06:14 -05:00
Sean Dague
8f90f765fa clean mysql better
The purge that was previously removed was actually kind of important
to burning mysql back down to a stateless zero point. Bring this back
with the addition of doing it for mariadb as well.

Change-Id: If608db8731d9ddfb2440a37387409798619b163c
2015-01-14 12:25:14 -05:00
Moshe Levi
d97d2cb017 fix mysql clean on ubuntu
Closes-Bug: 1409902
Change-Id: I72900eb2c7f3c6f66d829b10e9bb73b6d186da98
2015-01-13 15:02:19 +02:00
Ihar Hrachyshka
157c84b876 Enforce UTF-8 encoding when creating databases
We use InnoDB everywhere, so there should be no issues with long unicode
keys. Dropped charset parameter for recreate_database since it's not
needed anymore.

Change-Id: Ib768402a9337c918309030a92ab81da17269f4f6
2015-01-05 15:57:23 +01:00
Sean Dague
1cbb5d3f0c Clean database ubuntu/RHEL/SUSE use uninstall_package and clean /var/lib/mysql
Change-Id: I72c9df06903e536320a807fae8b817379e448444
Related-Bug: #1395776
2014-12-15 10:19:42 -05:00
Sean Dague
e263c82e48 add shebang lines to all lib files
With gerrit 2.8, and the new change screen, this will trigger syntax
highlighting in gerrit. Thus making reviewing code a lot nicer.

Change-Id: Id238748417ffab53e02d59413dba66f61e724383
2014-12-10 11:28:05 -05:00
pcrews
6de7dba863 Return mysql-server to default for all ubuntu installs
Change: I97cbede806e5c00363c7174fa1e9f286de96aab6 altered
the logic in lib/databases/mysql from installing mariadb only if
one was using rhel7 to installing it by default unless one was using
rhel6.

Change: Iae9a7a1c09f1fc83573c3926b3470955c244c401
Attempted to fix this, but only does so for precise.

Unless mariadb has become the default, I strongly prefer to keep
standard mysql as the database on ubuntu (precise or otherwise).

Closes-Bug: 1395776

Change-Id: I3afb89ae6d55405313b7219dd6daa24d9ca80f70
2014-11-24 08:04:39 -08:00
Akihiro Motoki
8f003ef9bf Use mysql in Ubuntu 12.04 (precise)
After commit 6d20f090, devstack does not work on Ubuntu 12.04
because Ubuntu 12.04 does not provide mariadb but devstack expects it.

Change-Id: Iae9a7a1c09f1fc83573c3926b3470955c244c401
2014-11-18 13:41:58 +09:00
Pavel Sedlák
6d20f09045 Initial support for Fedora 21
Initial support for Fedora 21.  Add f21 to the distribution lists for
rpms, ensure "mariadb" is used over "mysqld" and enable it in the
distro check, as we have several reports of it working.

Tested with [1]

[1] http://download.fedoraproject.org/pub/fedora/linux/releases/test/21-Beta/Cloud/Images/x86_64/Fedora-Cloud-Base-20141029-21_Beta.x86_64.qcow2

Change-Id: I97cbede806e5c00363c7174fa1e9f286de96aab6
2014-11-12 17:46:40 +01:00
Dean Troyer
3ef23bceec Clean up local variable usage - Backends
Combines cleanup for mysql, postgresql and rpc

Change-Id: I37b928a669146671c946fc1ccb8e3ef1a27a3891
2014-07-25 16:24:00 -05:00
Sean Dague
e56c137edb make mysql run in strict mode
this might obviate the need for postgresql in the gate, as it will
make mysql act more strict. Suggested by Monty during Atlanta
summit on test matrix.

Change-Id: Ibfcb0959159f3b84fe35a1198d481bff308da0d4
2014-06-03 05:57:30 -04:00
Attila Fazekas
3b53aebabd Fedora mysql log dir is /var/log/mariadb
/var/log/mysql does not exists on f20, f19, rhel7 by default,
but the /var/log/mariadb exists.

Change-Id: I0def3eaead491a00e6f8467db6b31502b9376188
2014-04-30 11:57:22 +02:00
Ralf Haferkamp
0526bb8222 Use iniset for changing my.conf and avoid deprecated config options
The mysql config file is using INI format so use the iniset function to
manipulate it. This change also rearranges the config updates a bit
allowing us to make mulitple changes in a single sudo call. This reduces
the number of required process forks, and the number of times the 'functions'
file needs to be sourced a bit.

The "log-slow-queries" option is deprecated since mysql 5.1.29 and got
removed with 5.6.x. Use the newer slow-query-log-file/slow-query-log
settings instead. They are available since 5.1.12. This fixes a problem
with running devstack with mysql-5.6, which is e.g. part of openSUSE
13.1.

Change-Id: Iea28bf05c664b5387d51dae1a63a780344623596
2014-04-10 14:03:14 +02:00
Tiago Mello
4376ae04df Clean /etc/mysql when calling clean.sh
The clean.sh script should also remove the /etc/mysql
directory. It contains information from the old
devstack installation and may conflict with the further
one. apt-get purge does not remove it since the
directory is not empty.

Change-Id: I885345a2311851d8746abe42e44300ecd4f6e08a
2014-03-14 10:58:02 -03:00
Jenkins
ebd1321fcb Merge "Removes the dependence with aptitude" 2014-02-26 12:47:39 +00:00
Sahid Orentino Ferdjaoui
e9648276a6 Removes the dependence with aptitude
Removes the dependence with aptitude by replacing
the call of:
  aptitude purge -y ~npackage by apt_get purge -y package*

Change-Id: I08875ffad9dc6293047827666f02453a355b16ea
Closes-Bug: 1281410
2014-02-24 09:41:07 +01:00
Attila Fazekas
2d65059e72 Add RHEL7 beta support
RHEL7 still in beta status, so it will require the FORCE option,
until the GA release.

The main notable difference from another RHEL family members, it does
not have the mysql alias for the mariadb.

Change-Id: Ic90bb6c3dd9447fc80453c3dc1adb22cdfc6226f
2014-02-20 17:02:51 +01:00
Jeremy Stanley
c4f47345a5 Make MySQL query logging optional
* lib/databases/mysql: Wrap query log configuration in a check for a
ENABLE_QUERY_LOGGING variable.

* stackrc: Add the DATABASE_QUERY_LOGGING variable defaulted to True.

Change-Id: Iddf8538ad0a1e36e2c6944dc70315984026c8245
2014-01-31 20:39:10 +00:00
Adam Spiers
6a5aa7c6a2 Fix some Markdown formatting issues
Address miscellaneous issues with Markdown formatting in comments which
are consumed by shocco when generating the online documentation.

Change-Id: I953075cdbddbf1f119c6c7e35f039e2e54b79078
2013-10-24 17:38:19 +01:00