openstack
/
tripleo-heat-templates
Code Issues Proposed changes

Create docker-puppet.sh only once to avoid ETXTBSY errors 

We currently create /var/lib/docker-puppet/docker-puppet.sh
inside the mp_puppet_config() function which then gets
invoked in parallel via the following:

  p = multiprocessing.Pool(process_count)
  returncodes = list(p.map(mp_puppet_config, process_map))

This is problematic because we have the following potential race:
1) Process A opens /var/lib/docker-puppet/docker-puppet.sh for writing
2) Process B runs docker run and has the following bind mount:
   /var/lib/docker-puppet/docker-puppet.sh:/var/lib/docker-puppet/docker-puppet.sh:z
3) Process B will fail because an exec of a file being written to
   will return ETXTBSY

The deployment can fail due to the above with the following error:
[root@overcloud-controller-2 ~]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a401108cd827 192.168.24.1:8787/tripleoqueens/centos-binary-glance-api:current-tripleo-rdo "/var/lib/docker-p..." 19 minutes ago Exited (1) 19 minutes ago docker-puppet-glance_api
[root@overcloud-controller-2 ~]# docker logs docker-puppet-glance_api
standard_init_linux.go:178: exec user process caused "text file busy"

Since /var/lib/docker-puppet/docker-puppet.sh never changes
there is really no need to create it multiple times. Let's just
create it once before spawning the multiple docker run commands
so we avoid any ETXTBSY errors.

Ran 20 successful deployments in sequence with this change applied.

Change-Id: I16b19488ce9f1411273459576db76d16b318dacb
Closes-Bug: #1760787
(cherry picked from commit 1037786b6c)
This commit is contained in:
Michele Baldessari 2018-05-31 12:30:20 +02:00
parent 3570a94a69
commit 96ae89f07c
1 changed files with 58 additions and 58 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

26
docker/docker-puppet.py
View File

@ -201,19 +201,8 @@ for service in (json_data or []):
log.info('Service compilation completed.')
def mp_puppet_config((config_volume, puppet_tags, manifest, config_image, volumes)):
log = get_logger()
log.info('Starting configuration of %s using image %s' % (config_volume,
config_image))
log.debug('config_volume %s' % config_volume)
log.debug('puppet_tags %s' % puppet_tags)
log.debug('manifest %s' % manifest)
log.debug('config_image %s' % config_image)
log.debug('volumes %s' % volumes)
sh_script = '/var/lib/docker-puppet/docker-puppet.sh'
with open(sh_script, 'w') as script_file:
sh_script = '/var/lib/docker-puppet/docker-puppet.sh'
with open(sh_script, 'w') as script_file:
os.chmod(script_file.name, 0755)
script_file.write("""#!/bin/bash
set -ex
@ -269,6 +258,17 @@ def mp_puppet_config((config_volume, puppet_tags, manifest, config_image, volume
fi
""")
def mp_puppet_config((config_volume, puppet_tags, manifest, config_image, volumes)):
log = get_logger()
log.info('Starting configuration of %s using image %s' % (config_volume,
config_image))
log.debug('config_volume %s' % config_volume)
log.debug('puppet_tags %s' % puppet_tags)
log.debug('manifest %s' % manifest)
log.debug('config_image %s' % config_image)
log.debug('volumes %s' % volumes)
with tempfile.NamedTemporaryFile() as tmp_man:
with open(tmp_man.name, 'w') as man_file:
man_file.write('include ::tripleo::packages\n')