Commit Graph

46 Commits

Author SHA1 Message Date
Mark Goddard
01050dc01f Support configuration of Docker client timeout
Adds support for configuration of the Docker client timeout via
'docker_client_timeout'.

This change also increases the default timeout to 120 seconds, as we
sometimes see timeouts in CI and heavily loaded or underpowered
environments. Increasing 'docker_client_timeout' further may be helpful
in cases where Docker reports 'Read timed out'.

Change-Id: I73745771078cb2c0ebae2b1d87ba2c4c12958d82
Closes-Bug: #1809844
2019-11-28 10:10:00 +00:00
Mark Goddard
64d07c0b7f Attempt to pull image before stopping and removing container
* Deploy services using kolla-ansible deploy
* Reconfigure the image for one or more services to use an invalid
* config
* Deploy/reconfigure services using kolla-ansible reconfigure

The invalid config could be a wrong docker registry, wrong image name,
wrong tag, etc.

The restart handler for the service fails, and the old container is
left running.

The restart handler for the service fails, and the old container is
stopped and removed. This leaves the service in a broken state.

This change fixes the issue by pulling the image if necessary prior to
stopping and removing the container.

Change-Id: I85b2a1b224d4c4d85c32c4922a2cd2c41171a1dc
Closes-Bug: #1852572
2019-11-14 15:08:09 +00:00
Michal Nasiadka
4180bee020 Use fluentd image labels
In order to orchestrate smooth transition to fluentd 0.14.x
aka 1.0 stable branch aka td-agent 3
from td-agent repository - use image labels (fluentd_version
and fluentd_binary).

Depends-On: https://review.opendev.org/676411
Change-Id: Iab8518c34ef876056c6abcdb5f2e9fc9f1f7dbdd
2019-08-22 12:36:51 +00:00
Radosław Piliszek
6a737b1968 Fix handling of docker restart policy
Docker has no restart policy named 'never'. It has 'no'.
This has bitten us already (see [1]) and might bite us again whenever
we want to change the restart policy to 'no'.

This patch makes our docker integration honor all valid restart policies
and only valid restart policies.
All relevant docker restart policy usages are patched as well.

I added some FIXMEs around which are relevant to kolla-ansible docker
integration. They are not fixed in here to not alter behavior.

[1] https://review.opendev.org/667363

Change-Id: I1c9764fb9bbda08a71186091aced67433ad4e3d6
Signed-off-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>
2019-07-18 13:39:06 +00:00
Mark Goddard
d2ae42ce6d Add support for idempotent container stop and removal
This is useful when removing a container that is no longer supported.

Change-Id: I08d79ce7dd2f3d11e466930de85412017cd5f747
2019-06-07 13:50:01 +01:00
Victor Coutellier
3bb4c19981 Fix kolla-docker possible undefined variable
It is possible to reference undefined variable in kolla-docker module if
DockerWorker object initialization fail, so the current behaviour will
crash the playbook with the unwanted error message :

UnboundLocalError: local variable 'dw' referenced before assignment

Change-Id: Ic8d26b11f93255220888b5406f8ab4a6f81736c2
Closes-Bug: #1819361
2019-03-10 19:13:00 +01:00
binhong.hua
3d3f5f1613 add ulimit support for kolla_docker
By default, docker containers inherit ulimit from limits of docker
deamon. On CentOS 7, docker daemon default NOFILE is 1048576.
It can found in /usr/lib/systemd/system/docker.service.
The big limit will cause many problem. we should control it in
production environment.

Change-Id: Iab962446a94ef092977728259d9818b86cfa7f68
2019-01-24 10:54:14 +08:00
Mark Goddard
365bb5177d Create cells before starting nova services
Nova services may reasonably expect cell databases to exist when they
start. The current cell setup tasks in kolla run after the nova
containers have started, meaning that cells may or may not exist in the
database when they start, depending on timing. In particular, we are
seeing issues in kolla CI currently with jobs timing out waiting for
nova compute services to start. The following error is seen in the nova
logs of these jobs, which may or may not be relevant:

No cells are configured, unable to continue

This change creates the cell0 and cell1 databases prior to starting nova
services.

In order to do this, we must create new containers in which to run the
nova-manage commands, because the nova-api container may not yet exist.
This required adding support to the kolla_docker module for specifying a
command for the container to run that overrides the image's command.

We also add the standard output and error to the module's result when a
non-detached container is run. A secondary benefit of this is that the
output of bootstrap containers is now displayed in the Ansible output if
the bootstrapping command fails, which will help with debugging.

Change-Id: I2c1e991064f9f588f398ccbabda94f69dc285e61
Closes-Bug: #1808575
2018-12-14 19:26:42 +00:00
Eduardo Gonzalez
846c15d8e1 Allow set tty for containers
This change adds support to comfigure tty,
it was enabled by default but a recent patch
removed it. Some services such as Karaf in opendaylight
requires a TTY during startup.

Closes-Bug: #1806662
Change-Id: Ia4335523b727d0e45505cbb1efb40ccf04c27db7
2018-12-05 11:44:37 +00:00
Rabi Mishra
48bae90e8f Don't allocate pseudo-TTY when creating containers
With a pseudo terminal, service is not treated as a daemon
and signals would not work as expected.

Change-Id: I16aa29a7924df51659d973a81d8005ae3d86f57b
Related-Bug: #1799642
2018-11-03 22:35:20 +05:30
Eduardo Gonzalez
fffdc6b3e5 Remove_images in kolla_docker module
Add support to remove images from kolla_docker
ansible module.

Change-Id: Ib6bcb98b5b295a2d590df3013188913d1f7f3584
2018-09-14 17:14:25 +02:00
Lakshmi Prasanna Goutham Pratapa
76210a2d85 Apply Resource Constraints to Openstack Services
This commit is to apply resource-constraints only to few OpenStack services.
Commit to apply constraints to other services will be made in coming commits.

Partially-Implements: blueprint resource-constraints

Change-Id: Icafa54baca24d2de64238222a5677b9d8b90e2aa
2018-07-25 17:05:04 +01:00
Lakshmi Prasanna Goutham Pratapa
297eb5e88f Add resource Constraints to Nova containers.
This commit will constrain the dimensions of service `Nova`
and sub-containers deployed along with it.

A user can give the dimension values in `/etc/kolla/globals.yml`
the data-types just like stated in this commit.

Reference-Docs:
https://docs.docker.com/config/containers/resource_constraints/

Added Test-cases for the same.

Partially-Implements: blueprint resource-constraints
Change-Id: I6458d8fb7b26a6e7c3a9fd0d674d9cf129b0bf5d
2018-07-12 23:24:05 +05:30
wangwei
193ce726c2 Fix ipc_mode comparison in the latest docker
In old docker, if you do not specify ipc_mode, the default value is empty,
but in the latest docker, such as 17.09.0, if not specified, the default
is "IpcMode": "shareable", which will cause all containers to be deleted
and re-create when to redeploy or upgrade. This commit solves the
problem.

Change-Id: Ia8269b9c8066880e4aee23d6fdea8d9c04c41e44
Closes-Bug: #1747586
2018-04-06 13:35:43 +00:00
Eduardo Gonzalez
448a10df6c Add container state check in kolla_docker
Missing container status check in recreate_or_restart_container,
this causes if the container is not running (kolla-ansible stop),
to not be started with deploy/reconfigure/upgrade if any other param
changes.

Change-Id: I5cff5f367e963ba8b1807ec46469da817e40e468
Closes-Bug: #1714015
2018-01-26 07:30:11 +00:00
Jenkins
7e6fb5e308 Merge "Remove unnecessary setUp function in testcase" 2017-08-02 14:59:59 +00:00
Jeffrey Zhang
7f2ade5b5a Stop and start container rather than use restart directly
Docker has an issue[0] when restart container. But stop then start
works.

[0] https://github.com/moby/moby/issues/29704

Change-Id: If0a9c0c257cd72209be8e138a1f0b8871500e089
Closes-Bug: #1707097
2017-07-28 17:26:27 +08:00
Jenkins
cd20a68635 Merge "Remove warning during kolla_docker execution" 2017-07-24 10:42:00 +00:00
Bertrand Lallau
2d424ffb14 Remove warning during kolla_docker execution
Ansible check if modules parameters are named like
%password% and allow to hidden log param in this case.
This requires adding "no_log" parameters.
This patch just add "no_log" param in order to avoid
this warning.

Change-Id: I9c1df1093e0fd101090292d6e8bf3527f99aeb17
Closes-Bug: #1702244
2017-07-14 18:43:45 +02:00
Hongbin Lu
3daeea37ea Upgrade from docker-py to docker
The pypi package 'docker-py' [1] has been renamed to 'docker' [2].
It is better to move to the new 'docker' package because the old
package will be deprecated and all the new features will go into
the new package only.

Package 'docker' has been added to requirements [3]. The old
package 'docker-py' is still allowed to be in the global requirements
during the transition period but it should be removed after all or
most of the projects finsih the migration.

[1] https://pypi.python.org/pypi/docker-py
[2] https://pypi.python.org/pypi/docker
[3] https://review.openstack.org/#/c/423715/

Change-Id: Ibcd5a57a1fbf55dcc5a690e41f20917f95b63da0
2017-07-10 14:19:28 +00:00
lingyongxu
18449a2231 Remove unnecessary setUp function in testcase
In testcase, setUp will be called automatically. This patch used to
remove setUp functions that do nothing. Besides, it will keep code clean.

Change-Id: I60e368dc066af30ce2e53bf7e4cc7bba69387cd8
2017-06-08 11:32:10 +08:00
Jenkins
a8433495dd Merge "Add graceful_timeout argument to kolla_docker" 2017-05-10 08:40:36 +00:00
Duong Ha-Quang
d929359550 Add graceful_timeout argument to kolla_docker
Currently, when stoping/restarting container, Kolla uses default timeout value
between SIGTERM and SIGKILL provided by docker which is 10 sec. But some
services require more than it to finish graceful shutdown progress.

This patchset adds graceful_timeout to kolla_docker to override the default
one.

Partial Implements: bp signaling-to-container

Change-Id: Ica0b48a53c650cc23dfa1955027d2cf936a5932f
2017-04-13 16:53:14 +07:00
Mohammed Naser
27d675ea13
Switch kolla_docker to rely on SHA256 for image changes
At the moment, the process to determine if an image has
changed or not relies on the Docker API which depending
on the Docker release server can return different results.

This patch addresses this issue by grabbing the SHA256
of the image before pulling (defaulting to None if it does
not exist) and then comparing it after the pull is complete
which should always be successful at determining if the
image did change or not.

The test for unknown status images is removed because this
is not a possible scenario as we do not rely on status
anymore except for failures (which are still tested).

Change-Id: Ia60a7f34420b02f50597dddb96a4c36ff3996612
Closes-Bug: #1668059
2017-04-06 16:34:01 -04:00
Abel Lopez
8b90fb6d47
Change python interpreter
There is inconsistent use of either `/usr/bin/python` or
`/usr/bin/env python`. This makes for unexpected results when a
user might be using a virtualenv.

Change-Id: Ibb030f920a8869f9113ade70b66a921cc815060d
2017-02-16 13:39:36 -08:00
Jenkins
6c33871071 Merge "Create a new container when restart a container when necessary" 2017-02-04 06:09:11 +00:00
Jeffrey Zhang
a3acd2c439 Fix the pid_mode and ipc_mode value check for kolla_docker module
bypass_checks is disabled by Ib2e19794bbe804470a880253d5870254041358c8 .
But we use pid_mode: "{{ service.pid_mode | default('') }}" pattern in
the yaml file. whereas pid_mode only support "host" value, this patch
add '' as a choice of these two parameters.

Change-Id: Ib83ed5e437ca868f0e42ce740ad1125d2fa48ff4
2017-02-03 15:56:10 +08:00
Jeffrey Zhang
869abbbed4 Create a new container when restart a container when necessary
Recreate and start a new container when container parameter is changed
in COPY_ALWAYS strategy.

Change-Id: I1e45c0dc34a93a18b664109823d6fd5b88c331a2
Closes-Bug: #1659799
2017-02-02 17:08:38 +08:00
zhuzeyu
42e7b747e4 Use required_if in kolla_docker.py
Replace the original code with required_if when Ansible 2.0 lands

Change-Id: Ib2e19794bbe804470a880253d5870254041358c8
2017-01-23 12:07:13 +08:00
Jenkins
3b2fcf7674 Merge "Fix python3 compatibility in kolla_docker" 2017-01-05 10:07:44 +00:00
Jeffrey Zhang
d37da2cfa9 Optimize reconfigure action for glance
Partically-implements: blueprint better-reconfigure
Change-Id: I89e30e8b87f24a621c521d915842a4af0042d6fe
2017-01-02 10:58:43 +08:00
Paul Bourke
17ec0f62f4 Fix python3 compatibility in kolla_docker
When using stream=True with docker-py it returns byte arrays rather than
strings.

Change-Id: I8eb6707ba2b122cf779135173abafe874b7223d2
Closes-Bug: #1643740
2016-11-25 10:53:36 +00:00
maniram477
6275f7090b Corrects typo "requried" in kolla_docker.py
The parameter "required=True" is mistyped as "requried=True"
in ansible/library/kolla_docker.py and tests/test_kolla_docker.py

Closes bug: 1640423

Change-Id: I64c333275b9eb3eff5b1c0fa5d550f478e68020d
2016-11-09 15:18:22 +05:30
Sajauddin Mohammad
3f134d0f53 Changed docker_restart_policy to unless-stopped
Current implementation has docker_restart_policy "always" and
docker_restart_policy_retry:"10" which should be mutually exclusive.
"unless-stopped" will restart the containers on any exit state
but not start on daemon startup if container was put to stopped state before.
Closes-Bug: #1621187

Change-Id: I4d881cd123a55625121b7a9047385e9b54e2e129
2016-09-26 04:32:28 -04:00
Jenkins
9d37b789ac Merge "Added Test case for env and volume comp" 2016-08-23 09:48:25 +00:00
Jenkins
3cec06ea72 Merge "Added Test case for attribute comp" 2016-08-23 09:48:18 +00:00
MD NADEEM
b8de1f895b Added Test case for env and volume comp
Change-Id: Ie98072ded6fee96b4e8c43cfe5124ead8e73e885
Implements: blueprint ansible-unit-tests
2016-08-11 06:48:34 +00:00
MD NADEEM
0c638e5b3e Added Test case for attribute comp
Change-Id: Ib9f3773d86fd00941f8dfb6f9b1019b01044cad3
Partially-Implements: blueprint ansible-unit-tests
2016-08-02 03:35:10 +00:00
Jeffrey Zhang
ae34973d35 Fix the kolla_docker issue with docker 1.12
After docker 1.12, the RepoTags will be None rather than [] in the
image list API. This PS will handle the both case.

Change-Id: Ie2da44b44229c2f190550755b50876f607f9cc0c
Closes-Bug: #1608358
2016-08-01 13:41:05 +08:00
MD NADEEM
ef37687e35 Added unit test for volume operations
Change-Id: I7ad73a81559d8856bd8a41d24f274a19d5a9ff03
Partially-Implements: blueprint ansible-unit-tests
2016-07-14 11:07:31 +00:00
MD NADEEM
f5b04c97eb Added unit test for image operation
Change-Id: I44a5ba649d73f4ca2d48065625c8d67445576944
Partially-Implements: blueprint ansible-unit-tests
2016-06-28 10:57:47 +00:00
MD NADEEM
31e5f67e17 Add test for remove container, get state and get env
Change-Id: Id3abd50dbcdb21654d5f5bc344e960aff2c5d5d7
Partially-Implements: blueprint ansible-unit-tests
2016-06-27 16:38:30 +00:00
MD NADEEM
02e060e099 Add unit test for stop and restart container
Change-Id: Icd73eb6e2d5b4207142ef249c8995c98056ae1c4
Partially-Implements: blueprint ansible-unit-tests
2016-06-27 14:11:51 +05:30
MD NADEEM
0e459bb6d3 Added unit test for start container
This patch check two senario
first if  we choose to start a new container
with name say XXX, however in the env a container
already exist with XXX name and second to start a stopped container.

Change-Id: Ibd6c5fbcc6f37ea2481634ac2e6b3edb91ae7aab
Partially-Implements: blueprint ansible-unit-tests
2016-06-16 21:20:38 +05:30
MD NADEEM
45799699fe Added unit test for start container
Partially-Implements: blueprint ansible-unit-tests
Change-Id: I398f73f24ee5fa29dd11cd9bb8582cc6c195d007
2016-06-14 17:22:52 +00:00
MD NADEEM
2634a94e7a Added unit test for kolla_docker
Added unit test for create container.

Change-Id: If67025791e52581c8853da50623cc9547f4e9298
Partially-Implements: blueprint ansible-unit-tests
2016-06-12 13:32:24 +05:30