Add DockerPuppetProcessCount defaults to 3

docker-puppet.py is very aggressive about running concurrently.
It uses python multiprocessing to run multiple config generating
containers at once. This seems to work well in general, but
in some cases... perhaps when the registry is slow or under
heavy load can cause timeouts to occur. Lately I'm seeing
several 'container did not start before the specified timeout'
errors that always seem to occur when config files are generated
(docker-puppet.py is initially executed.

A couple of things:

 -when config files are generated this is the first time
  most of the containers are pulled to each host machine
  during deployment

 -docker-puppet.py runs many of these processes at once. Some
  of them run faster, other not.

 -docker daemon's pull limit defaults to 3. This would throttle
  the above a bit perhaps contributing the the likelyhood of a timeout.

One solution that seems to work for me is to set the PROCESS_COUNT
in docker-puppet.py to 3. As this matches docker daemon's default
it is probably safer at the cost of being slightly slower in some
cases.

Change-Id: I17feb3abd9d36fe7c95865a064502ce9902a074e
Closes-bug: #1713188
This commit is contained in:
Dan Prince 2017-08-25 23:01:24 -04:00
parent 451750c17b
commit 949d367dde
2 changed files with 7 additions and 0 deletions

View File

@ -28,6 +28,7 @@
environment:
NET_HOST: 'true'
DEBUG: '{{docker_puppet_debug|default(false)}}'
PROCESS_COUNT: '{{docker_puppet_process_count|default(3)}}'
when: step == "1"
changed_when: false
check_mode: no

View File

@ -44,6 +44,10 @@ parameters:
type: string
default: ''
description: Set to True to enable debug logging with docker-puppet.py
DockerPuppetProcessCount:
type: number
default: 3
description: Number of concurrent processes to use when running docker-puppet to generate config files.
ctlplane_service_ips:
type: json
@ -74,6 +78,7 @@ resources:
- name: update_identifier
- name: bootstrap_server_id
- name: docker_puppet_debug
- name: docker_puppet_process_count
config:
str_replace:
template: |
@ -273,6 +278,7 @@ resources:
update_identifier: {get_param: DeployIdentifier}
bootstrap_server_id: {get_param: [servers, {{primary_role_name}}, '0']}
docker_puppet_debug: {get_param: DockerPuppetDebug}
docker_puppet_process_count: {get_param: DockerPuppetProcessCount}
{% endfor %}
# END CONFIG STEPS