195 Commits

Author SHA1 Message Date
wu.chunyang
8ea76a145a Drop bionic support
drop the support of bionic.
ref:
https://governance.openstack.org/tc/reference/runtimes/zed.html

Change-Id: I6dd126d49196af30bd5578c6aefa90fff9f1d5c1
2022-08-12 06:51:00 +00:00
wu.chunyang
3285b6dc8a Increase guest image size to 5G
trovestack built image failed due to the less disk space.
this ps increases the image size from 3 to 5G
ref:
https://zuul.opendev.org/t/openstack/build/fedba359fbc8427c886ff2b2f0767d37/console

Change-Id: I7e48d74e2e03fe168a185949b162b3dfcc6fb7d5
2022-08-12 10:19:53 +08:00
wu.chunyang
2f8f2f8d92 Add ubuntu focal support for trove guest image
story: 2009996
task: 45072
Change-Id: Ib3613119dc136c8a0d96a27c52e0bc1874065775
2022-04-19 09:49:03 +08:00
wu.chunyang
4d8e55c5fe Add image build test on Centos8 stream
build image on Centos8 stream

Change-Id: I759d482430eaf0e9304986c9fad047cbd44c09ca
2022-04-17 16:53:52 +08:00
wu.chunyang
c0ad3f8c64 WIP: add guest image build check
This PR adds trove-guest-image-build check in ci.

Change-Id: If512cb7c094d14850555215d412b8e7b418c35ce
2022-04-13 22:51:20 +08:00
Lingxian Kong
1e4405c60b Fix trove guest agent logrotate
* The logrotate file owner should be root
* Log file path is /var/log/trove/trove-guestagent.log

Change-Id: I669fedc76aa79d7bfc1e4e64684adc4a89a3a15d
2021-09-30 22:27:53 +13:00
Lingxian Kong
4ce59ace17 Add package iptables-persistent inside guest instance
iptables-persistent is needed to persists the iptable settings inside
guest instance.

Change-Id: I96bb9f948dda22f50ec2a4fb5710ad9778e16b9d
2021-07-09 11:27:40 +12:00
Lingxian Kong
b050996b9f Use bridge network for db container
- Changed the network mode of database container to "bridge" and exposed
  the service ports.
- Use socket file to connect with the database.
- Upgrade the backup container image for postgressql.

Change-Id: Id5b119f8a474befc3a2cd6e061bbffc4ae5f7bb6
2021-06-22 19:10:14 +12:00
Lingxian Kong
1e04b269ca Stop using pip-and-virtualenv diskimage element
ALthough diskimage-builder will solve the issue eventually, but we
still prefer to put the package installation under trove's own control.

Story: 2008559
Task: 41697

Change-Id: Ie704db8233c4b797eeefa140c4a11427280dffd9
2021-01-27 11:43:16 +13:00
Zuul
774cb66671 Merge "Improve trove guest image build script" 2020-11-29 10:49:01 +00:00
Lingxian Kong
9dd8e31516 Remove the experimental claim of dev_mode false
I am using the guest image built by dev_mode=false for some time, it's stable.

Change-Id: Ic74d10647f82044038ea9a4d99cb9a0320459675
2020-11-27 11:30:38 +13:00
Lingxian Kong
a97ff84c19 Improve trove guest image build script
Also using ubuntu-minimal to decrease the image size a little bit, from 963M
to 815M in the test.

Change-Id: I6d70add380f4b61ebe89c72765612d8cb6127ccb
2020-11-26 14:17:57 +13:00
Lingxian Kong
da2da6969a Support --image-tags for trove-manager datastore_version_update command
image-tags was introduced in commit 1d24b65 but the implementation was
backward incompatible for the CLI `trove-manager datastore_version_update`.

This patch changes that by supporting an optional parameter `--image-tags`.

Change-Id: I583c296f15a453bdfadf10de5c678ac483cbd4dd
2020-11-26 11:06:55 +13:00
Lingxian Kong
33c0b64051 Add 'version' to datastore version
This patch is part of story 2008358 implementation.

1. DB schema change
2. trove-manage datastore_version_update subcommand change
3. API change for creating and listing datastore version.

Story: 2008358
Task: 41264

Change-Id: I8069e6f4b972497f2b8be540ea35047d5fb2f9a5
2020-11-17 22:10:56 +13:00
Lingxian Kong
d1af33f17b Support mysql 8.0
* MySQL 5.7 and MySQL 8.0 need different percona-xtrabackup package version.
  Added Percona XtraBackup 8 support for MySQL 8.x backup and restore.
* Construct different backup container image names for MySQL 5.7 and MySQL 8.0
  based on the default option value.
* Two docker images are uploaded for backup/restore:
  openstacktrove/db-backup-mysql5.7:1.0.0 and
  openstacktrove/db-backup-mysql8.0:1.0.0. Trove guest agent can automatically
  choose the approriate one based on the datastore version.
* Added option "secure-file-priv=NULL" in MySQL config template to fix
  https://github.com/docker-library/mysql/issues/541.
* Stop using IDENTIFIED BY in GRANT clause (also REVOKE). Starting with MySQL 8
  creating a user implicitly using the GRANT command is not supported.

Story: #2008275
Task: #41143

Change-Id: Ibdec63324b1b39ba9b8a38dbe529da17bbb06767
2020-10-23 23:50:39 +13:00
wangzihao
e954184693 Remove six usage and basestring check
Remove basestring check.
Remove six Replace the following items with Python 3 style code.

- six.string_types
- six.int2byte
- six.indexbytes
- six.add_metaclass
- six.StringIO
- six.text_type
- six.integer_types
- six.binary_type
- six.BytesIO
- six.reraise

Change-Id: I4fb9033d152963c504ceb4d5c4d08f934ee4accb
2020-10-16 10:40:22 +08:00
Lingxian Kong
1d659d0342 Fix the image CI job
Change-Id: I08a3a5873bf883c453e86e662296f7d0c7fbaf49
2020-10-10 00:13:22 +13:00
Lingxian Kong
1d24b65052 Image tags support in datastore version
Change-Id: I0e51d08515c121e3a7b0e82b6e4c4161bb4fbc4a
2020-10-09 11:43:48 +13:00
Lingxian Kong
7aa269583c Install pip3 for image build
Change-Id: I3117cc56aa0edf2f4345784291ae8294a8ffc07a
2020-09-22 00:01:38 +12:00
Zuul
768ec34dfe Merge "Use sudo when deleting image cache dir" 2020-09-04 07:14:16 +00:00
Zuul
488d8a71e2 Merge "Remove six.PY3" 2020-09-04 04:21:02 +00:00
Lingxian Kong
ec8a10e36b Use sudo when deleting image cache dir
Change-Id: I8a9fd906d6b70dfad1c38e751cadc004a5aeb13f
2020-09-04 13:58:56 +12:00
likui
5bf159de36 Remove six.PY3
The Python 2.7 Support has been dropped since Ussuri.
So remove hacking rules for compatibility between python 2 and 3.

Change-Id: Id3fd78b5d3e21579975ea0d0fc65e4f66ea4582e
2020-09-03 18:30:24 +08:00
Lingxian Kong
00a07f2423 Use pip3 instead of pip in trovestack
Change-Id: Icbbfaa07a699e1280b0d39dfbae87d7468cbb260
2020-09-02 10:11:12 +12:00
Lingxian Kong
9914bd6aad Allow to specify root password when building guest image
For testing purpose.

Change-Id: I8193f4262156ed9b0784666f5981ebd6df4d525b
2020-08-12 21:36:00 +12:00
Lingxian Kong
cf3e9a6e74 Support to rebuild instance by admin
Change-Id: I48f8c6f997daeb6d82ff62b3325277d86fee2147
2020-08-06 20:38:11 +12:00
Lingxian Kong
428e3644f8 Support image type for guest image building
Change-Id: I8d640c47f00446d7f912ece9892b17585cde3add
2020-06-12 22:41:23 +12:00
Hervé Beraud
816a7bca58 Stop to use the __future__ module.
The __future__ module [1] was used in this context to ensure compatibility
between python 2 and python 3.

We previously dropped the support of python 2.7 [2] and now we only support
python 3 so we don't need to continue to use this module and the imports
listed below.

Imports commonly used and their related PEPs:
- `division` is related to PEP 238 [3]
- `print_function` is related to PEP 3105 [4]
- `unicode_literals` is related to PEP 3112 [5]
- `with_statement` is related to PEP 343 [6]
- `absolute_import` is related to PEP 328 [7]

[1] https://docs.python.org/3/library/__future__.html
[2] https://governance.openstack.org/tc/goals/selected/ussuri/drop-py27.html
[3] https://www.python.org/dev/peps/pep-0238
[4] https://www.python.org/dev/peps/pep-3105
[5] https://www.python.org/dev/peps/pep-3112
[6] https://www.python.org/dev/peps/pep-0343
[7] https://www.python.org/dev/peps/pep-0328

Change-Id: I0adf51fb6a54c323c3d7a26ea62c23184a99933d
2020-06-02 21:06:08 +02:00
Lingxian Kong
e2b26fdd7d Trove doc and CI
Change-Id: I49bd6c824b837d22d9f3945a3e8c2343c831b352
2020-05-28 22:53:17 +12:00
Lingxian Kong
aa1d4d2246 Datastore containerization
Significant changes:

* Using docker image to install datastore.
* Datastore image is common to different datastores.
* Using backup docker image to do backup and restore.
* Support MariaDB replication
* Set most of the functional jobs as non-voting as nested
  virtualization is not supported in CI.

Change-Id: Ia9c97a63a961eebc336b70d28dc77638144c1834
2020-05-27 10:31:50 +12:00
Lingxian Kong
523d66e8fd Fix CI job
* Fix pep8: ambiguous variable name
* Install Python 3.7 for trove guest image. oslo.concurrency requires
  python 3.6 or newer. See
  https://bugs.launchpad.net/tripleo/+bug/1861803
* Mark tempest job non-voting temporarily because of some tempest bugs.

Change-Id: I6d316779cc7220a855ce187437056b667bbe1f75
2020-05-25 17:37:38 +12:00
Lingxian Kong
7d78c74158 Add python-troveclient in requirements.txt
Change-Id: I2bd0d81171df85c05423e19ab2e4afa82fcf880e
2020-03-15 20:57:39 +13:00
Lingxian Kong
602c4d42de Improve the function tests
- Execute test groups in serial to make sure no more than 2 database
  instance are created at the same time.
- Remove some unneccesary tests
- Remove unneeded datastore, e.g. 'Test_Datastore_1'
- Remove unsupported trovestack subcommands
- Move unsupported DIB elements to the 'deprecated-elements' folder
- Decrease default value of 'agent_call_high_timeout' to 5min
- Add initial_deplay for pooling task
- Use socket file to connect with database instead of using localhost
  IP

Change-Id: Ie5030a671fbeb453eafa6cbe04e08da7b52e33c9
2020-01-28 14:27:52 +13:00
Lingxian Kong
72524f9d85 Support PostgreSQL 12
Change-Id: Ic1d4982a479c2154d7ddcc807d863e2a37fea506
2019-12-18 22:13:32 +00:00
Lingxian Kong
4c8482095d Fix Trove periodic CI jobs
Change-Id: I11456aaeb2d417dbae6bdb0e51944aa122c521e0
2019-12-08 21:11:44 +00:00
Lingxian Kong
4c551947c9 Move the iptable change from trovestack to devstack
Change-Id: I6e10744f86f19f3c29babc07fda77936c95022ff
2019-12-08 18:50:42 +13:00
Zuul
94331620a3 Merge "Support pip3 and run on guest-agent service for redis" 2019-12-06 09:57:17 +00:00
Lingxian Kong
637c900b87 Mark CI job trove-scenario-mariadb-single voting
Fix the current CI issue as well.

Change-Id: I7daafa68a03a3bf88c3651b6cab8c58cf5256b29
2019-12-06 11:21:45 +13:00
Nguyen Van Trung
d5549f30cf Support pip3 and run on guest-agent service for redis
This patch will solve some following items:
- Support python3/ pip3
- Permit redis is running with guest-agent service

Tested with standalone, replicate and cluster successfully.

Change-Id: I81b72fae1c31ce2995232b52460d0ad0d6f894be
2019-12-05 09:35:12 +00:00
Nguyen Van Trung
1d314fdf91 Support pip3-virtualenv during image creation
This patch will auto-process with `pip-and-virtualenv`
element when use `guest-agent` in python3. We also can add
this element in disk-image-create cli which instead of
this patch to fix following issue:

   2019-09-12 09:19:58.674 | + python3 -m virtualenv /opt/guest-agent-venv
   2019-09-12 09:19:58.752 | /usr/bin/python3: No module named virtualenv

Change-Id: I21090320ba7396979cee31431de7198150dd763e
2019-12-05 09:34:52 +00:00
Lingxian Kong
11b0b8d6f2 Fix mariadb CI - trove-scenario-mariadb-single
- Trove supports MariaDB 10.4
- From MariaDB 10.3, Mariabackup is used instead of Percona XtraBackup
  for backup functionality
- Some log improvements

Change-Id: Ibaa6fd7273b98451097b32fb6b881008a236be9f
2019-12-05 07:48:26 +13:00
Lingxian Kong
8fc0b7695d Build reusable Trove guest image for dev
When building the guest image for dev_mode=true, the controller IP
address is injected to the image in order for the guest-agent to
download Trove code during initialization. As a result, we have to build
guest image each time in the Trove CI because the image build relies on
the devstack host IP address.

If we could remove the dependency, we can build the image(in
devmode=true) for a specific datastore once and use that image for all
related CI jobs, which could save some time for Trove CI.

Fix the current CI issue as well.

Change-Id: If23f4f179a6ab72cfb35e4c45d55142fedb76498
2019-12-05 00:45:56 +13:00
Lingxian Kong
d244c9d666 Support to specify branch when building the image
- Explicity specify branch name for building non-dev image
- Use branch name for building dev image in devstack
- Update the doc

Candidate backport for stable/train

Change-Id: I4d2fe38fac34c75b42234fa7a20aba51e6dc1c4e
2019-10-10 21:55:53 +13:00
Lingxian Kong
650794eaf9 Remove all the resources when the Nova VM creation failed
Backport candidate for stable/train

Story: 2006664
Task:  36926

Change-Id: If0991e0cef40ca78752fcd509d8438e90c9557bc
2019-10-09 12:03:59 +13:00
Lingxian Kong
32de39fa4d Use correct Swift credential for instance backup
DevStack is deploying Trove in service tenant model, for swift backed
instance backup, the swift container should be transparent to the end
users.

Story: 2006647
Task: 36883

Change-Id: I5859f4c9911fc2c129c8f23611c6607044fbc145
2019-10-02 22:15:48 +13:00
Zuul
d290514706 Merge "Some clean up" 2019-09-20 02:02:39 +00:00
Zuul
a67a6ed2ef Merge "Fix issue with pip2 command and update pip3 for mongodb" 2019-09-19 23:49:04 +00:00
Lingxian Kong
757a873963 Some clean up
- Remove the unused variables in trovestack.rc which leads to
  module not found error.
- Remove non-voting gate job

Change-Id: I583945addb6cb0aa442db1c9d9c6ce3cb57793f7
2019-09-20 08:51:21 +12:00
Lingxian Kong
0cdf8b20b6 Add image build experimental CI job
Story: 2006553
Task: 36637
Change-Id: I3d909082fb887b24ed25c93e11792031d5053b1e
2019-09-17 22:23:43 +12:00
Nguyen Van Trung
38efc97251 Fix issue with pip2 command and update pip3 for mongodb
This patch will fix following issue logs:

   2019-09-02 15:11:13.121 | dib-run-parts Running /tmp/in_target.d/install.d/25-trove-mongo-dep
   2019-09-02 15:11:13.124 | + pip2 install pymongo
   2019-09-02 15:11:13.125 | /tmp/in_target.d/install.d/25-trove-mongo-dep: line 9: pip2: command not found

And move on pip3.

Change-Id: I7cf16820091a0f977b41e3222abb866a9ae66166
2019-09-12 10:11:30 +00:00