performa/shaker
Code Issues Proposed changes

Add support for heat stack related configurations

Browse Source 
This commit enhances Shaker to allow:
  - specifying a custom stack name, instead of always using a uniquely
    generated one. This helps with tracking test stacks in CICD and/or
    allowing users to be more descriptive when looking at running heat
    stacks
  - reusing an existing test stack, rather than re-stacking. This
    is to allow the following scenario: Execute scenario A (setting
    cleanup_on_error = false) to establish baseline data. Perform some
    update in Openstack (config change, cert rotation, etc). Reuse the
    existing VMs from the baseline test to re-execute scenario A to
    help ensure the update had no impact.

Change-Id: Ifbdd332a44ca54f06cc81d9018ea5cea26c32416
This commit is contained in:
Oded Le'Sage 2019-08-15 11:26:42 -05:00
parent ecd88a3967
commit 8385ae1b97
11 changed files with 183 additions and 39 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
doc/source/tools/shaker-agent.txt
View File

@ -18,9 +18,11 @@ optional arguments:
over-ridden. The set is parsed after the file(s)
specified via previous --config-file, arguments hence
over-ridden options in the directory take precedence.
This option must be set from the command-line.
--config-file PATH Path to a config file to use. Multiple config files
can be specified, with values in later files taking
precedence. Defaults to None.
precedence. Defaults to None. This option must be set
from the command-line.
--debug, -d If set to true, the logging level will be set to DEBUG
instead of the default INFO level.
--log-config-append PATH, --log-config PATH, --log_config PATH
@ -31,7 +33,7 @@ optional arguments:
logging configuration files are used then all logging
configuration is set in the configuration file and
other logging configuration options are ignored (for
example, logging_context_format_string).
example, log-date-format).
--log-date-format DATE_FORMAT
Defines the format string for %(asctime)s in log
records. Default: None . This option is ignored if

22
doc/source/tools/shaker-all-in-one.txt
View File

@ -33,11 +33,12 @@ usage: shaker-all-in-one [-h] [--agent-join-timeout AGENT_JOIN_TIMEOUT]
[--os-username <auth-username>] [--output OUTPUT]
[--polling-interval POLLING_INTERVAL]
[--report REPORT] [--report-template REPORT_TEMPLATE]
[--reuse-stack-name REUSE_STACK_NAME]
[--scenario SCENARIO]
[--scenario-availability-zone SCENARIO_AVAILABILITY_ZONE]
[--scenario-compute-nodes SCENARIO_COMPUTE_NODES]
[--server-endpoint SERVER_ENDPOINT]
[--subunit SUBUNIT]
[--stack-name STACK_NAME] [--subunit SUBUNIT]
[--syslog-log-facility SYSLOG_LOG_FACILITY]
[--use-journal] [--use-json] [--use-syslog]
[--watch-log-file]
@ -67,9 +68,11 @@ optional arguments:
over-ridden. The set is parsed after the file(s)
specified via previous --config-file, arguments hence
over-ridden options in the directory take precedence.
This option must be set from the command-line.
--config-file PATH Path to a config file to use. Multiple config files
can be specified, with values in later files taking
precedence. Defaults to None.
precedence. Defaults to None. This option must be set
from the command-line.
--custom-user-opts CUSTOM_USER_OPTS
Set custom user option parameters for the scenario.
The value is specified in YAML, e.g. custom_user_opts
@ -127,7 +130,7 @@ optional arguments:
logging configuration files are used then all logging
configuration is set in the configuration file and
other logging configuration options are ignored (for
example, logging_context_format_string).
example, log-date-format).
--log-date-format DATE_FORMAT
Defines the format string for %(asctime)s in log
records. Default: None . This option is ignored if
@ -205,10 +208,16 @@ optional arguments:
Template for report. Can be a file name or one of
aliases: "interactive", "json". Defaults to
"interactive".
--reuse-stack-name REUSE_STACK_NAME
Name of an existing Shaker heat stack to reuse. The
default is to not reuse an existing stack. Caution
should be taken to only reuse stacks meant for a
specific scenario. Also certain configs e.g. image-
name, flavor-name, stack-name, etc will be ignored
when reusing an existing stack.
--scenario SCENARIO Comma-separated list of scenarios to play. Each entity
can be a file name or one of aliases:
"misc/instance_metadata", "misc/static_agent",
"misc/static_agents_pair",
"misc/instance_metadata",
"openstack/cross_az/full_l2",
"openstack/cross_az/full_l3_east_west",
"openstack/cross_az/full_l3_north_south",
@ -248,6 +257,9 @@ optional arguments:
--server-endpoint SERVER_ENDPOINT
Address for server connections (host:port), defaults
to env[SHAKER_SERVER_ENDPOINT].
--stack-name STACK_NAME
Name of test heat stack. The default is a uniquely
generated name.
--subunit SUBUNIT Subunit stream file name, defaults to
env[SHAKER_SUBUNIT].
--syslog-log-facility SYSLOG_LOG_FACILITY

18
doc/source/tools/shaker-cleanup.txt
View File

@ -19,6 +19,8 @@ usage: shaker-cleanup [-h] [--cleanup] [--cleanup-on-error] [--config-dir DIR]
[--os-tenant-name <auth-tenant-name>]
[--os-user-domain-name <auth-user-domain-name>]
[--os-username <auth-username>]
[--reuse-stack-name REUSE_STACK_NAME]
[--stack-name STACK_NAME]
[--syslog-log-facility SYSLOG_LOG_FACILITY]
[--use-journal] [--use-json] [--use-syslog]
[--watch-log-file]
@ -34,9 +36,11 @@ optional arguments:
over-ridden. The set is parsed after the file(s)
specified via previous --config-file, arguments hence
over-ridden options in the directory take precedence.
This option must be set from the command-line.
--config-file PATH Path to a config file to use. Multiple config files
can be specified, with values in later files taking
precedence. Defaults to None.
precedence. Defaults to None. This option must be set
from the command-line.
--debug, -d If set to true, the logging level will be set to DEBUG
instead of the default INFO level.
--dns-nameservers DNS_NAMESERVERS
@ -61,7 +65,7 @@ optional arguments:
logging configuration files are used then all logging
configuration is set in the configuration file and
other logging configuration options are ignored (for
example, logging_context_format_string).
example, log-date-format).
--log-date-format DATE_FORMAT
Defines the format string for %(asctime)s in log
records. Default: None . This option is ignored if
@ -120,6 +124,16 @@ optional arguments:
env[OS_USER_DOMAIN_NAME].
--os-username <auth-username>
Authentication username, defaults to env[OS_USERNAME].
--reuse-stack-name REUSE_STACK_NAME
Name of an existing Shaker heat stack to reuse. The
default is to not reuse an existing stack. Caution
should be taken to only reuse stacks meant for a
specific scenario. Also certain configs e.g. image-
name, flavor-name, stack-name, etc will be ignored
when reusing an existing stack.
--stack-name STACK_NAME
Name of test heat stack. The default is a uniquely
generated name.
--syslog-log-facility SYSLOG_LOG_FACILITY
Syslog facility to receive log lines. This option is
ignored if log_config_append is set.

18
doc/source/tools/shaker-image-builder.txt
View File

@ -28,6 +28,8 @@ usage: shaker-image-builder [-h] [--cleanup-on-error] [--config-dir DIR]
[--os-tenant-name <auth-tenant-name>]
[--os-user-domain-name <auth-user-domain-name>]
[--os-username <auth-username>]
[--reuse-stack-name REUSE_STACK_NAME]
[--stack-name STACK_NAME]
[--syslog-log-facility SYSLOG_LOG_FACILITY]
[--use-journal] [--use-json] [--use-syslog]
[--watch-log-file]
@ -42,9 +44,11 @@ optional arguments:
over-ridden. The set is parsed after the file(s)
specified via previous --config-file, arguments hence
over-ridden options in the directory take precedence.
This option must be set from the command-line.
--config-file PATH Path to a config file to use. Multiple config files
can be specified, with values in later files taking
precedence. Defaults to None.
precedence. Defaults to None. This option must be set
from the command-line.
--debug, -d If set to true, the logging level will be set to DEBUG
instead of the default INFO level.
--dns-nameservers DNS_NAMESERVERS
@ -94,7 +98,7 @@ optional arguments:
logging configuration files are used then all logging
configuration is set in the configuration file and
other logging configuration options are ignored (for
example, logging_context_format_string).
example, log-date-format).
--log-date-format DATE_FORMAT
Defines the format string for %(asctime)s in log
records. Default: None . This option is ignored if
@ -152,6 +156,16 @@ optional arguments:
env[OS_USER_DOMAIN_NAME].
--os-username <auth-username>
Authentication username, defaults to env[OS_USERNAME].
--reuse-stack-name REUSE_STACK_NAME
Name of an existing Shaker heat stack to reuse. The
default is to not reuse an existing stack. Caution
should be taken to only reuse stacks meant for a
specific scenario. Also certain configs e.g. image-
name, flavor-name, stack-name, etc will be ignored
when reusing an existing stack.
--stack-name STACK_NAME
Name of test heat stack. The default is a uniquely
generated name.
--syslog-log-facility SYSLOG_LOG_FACILITY
Syslog facility to receive log lines. This option is
ignored if log_config_append is set.

6
doc/source/tools/shaker-report.txt
View File

@ -20,9 +20,11 @@ optional arguments:
over-ridden. The set is parsed after the file(s)
specified via previous --config-file, arguments hence
over-ridden options in the directory take precedence.
This option must be set from the command-line.
--config-file PATH Path to a config file to use. Multiple config files
can be specified, with values in later files taking
precedence. Defaults to None.
precedence. Defaults to None. This option must be set
from the command-line.
--debug, -d If set to true, the logging level will be set to DEBUG
instead of the default INFO level.
--input INPUT File or list of files to read test results from,
@ -35,7 +37,7 @@ optional arguments:
logging configuration files are used then all logging
configuration is set in the configuration file and
other logging configuration options are ignored (for
example, logging_context_format_string).
example, log-date-format).
--log-date-format DATE_FORMAT
Defines the format string for %(asctime)s in log
records. Default: None . This option is ignored if

9
doc/source/tools/shaker-spot.txt
View File

@ -27,9 +27,11 @@ optional arguments:
over-ridden. The set is parsed after the file(s)
specified via previous --config-file, arguments hence
over-ridden options in the directory take precedence.
This option must be set from the command-line.
--config-file PATH Path to a config file to use. Multiple config files
can be specified, with values in later files taking
precedence. Defaults to None.
precedence. Defaults to None. This option must be set
from the command-line.
--custom-user-opts CUSTOM_USER_OPTS
Set custom user option parameters for the scenario.
The value is specified in YAML, e.g. custom_user_opts
@ -48,7 +50,7 @@ optional arguments:
logging configuration files are used then all logging
configuration is set in the configuration file and
other logging configuration options are ignored (for
example, logging_context_format_string).
example, log-date-format).
--log-date-format DATE_FORMAT
Defines the format string for %(asctime)s in log
records. Default: None . This option is ignored if
@ -86,8 +88,7 @@ optional arguments:
"interactive".
--scenario SCENARIO Comma-separated list of scenarios to play. Each entity
can be a file name or one of aliases:
"misc/instance_metadata", "misc/static_agent",
"misc/static_agents_pair",
"misc/instance_metadata",
"openstack/cross_az/full_l2",
"openstack/cross_az/full_l3_east_west",
"openstack/cross_az/full_l3_north_south",

28
doc/source/tools/shaker.txt
View File

@ -22,12 +22,13 @@ usage: shaker [-h] [--agent-join-timeout AGENT_JOIN_TIMEOUT]
[--os-user-domain-name <auth-user-domain-name>]
[--os-username <auth-username>] [--output OUTPUT]
[--polling-interval POLLING_INTERVAL] [--report REPORT]
[--report-template REPORT_TEMPLATE] [--scenario SCENARIO]
[--report-template REPORT_TEMPLATE]
[--reuse-stack-name REUSE_STACK_NAME] [--scenario SCENARIO]
[--scenario-availability-zone SCENARIO_AVAILABILITY_ZONE]
[--scenario-compute-nodes SCENARIO_COMPUTE_NODES]
[--server-endpoint SERVER_ENDPOINT] [--subunit SUBUNIT]
[--syslog-log-facility SYSLOG_LOG_FACILITY] [--use-journal]
[--use-json] [--use-syslog] [--watch-log-file]
[--server-endpoint SERVER_ENDPOINT] [--stack-name STACK_NAME]
[--subunit SUBUNIT] [--syslog-log-facility SYSLOG_LOG_FACILITY]
[--use-journal] [--use-json] [--use-syslog] [--watch-log-file]
optional arguments:
-h, --help show this help message and exit
@ -53,9 +54,11 @@ optional arguments:
over-ridden. The set is parsed after the file(s)
specified via previous --config-file, arguments hence
over-ridden options in the directory take precedence.
This option must be set from the command-line.
--config-file PATH Path to a config file to use. Multiple config files
can be specified, with values in later files taking
precedence. Defaults to None.
precedence. Defaults to None. This option must be set
from the command-line.
--custom-user-opts CUSTOM_USER_OPTS
Set custom user option parameters for the scenario.
The value is specified in YAML, e.g. custom_user_opts
@ -88,7 +91,7 @@ optional arguments:
logging configuration files are used then all logging
configuration is set in the configuration file and
other logging configuration options are ignored (for
example, logging_context_format_string).
example, log-date-format).
--log-date-format DATE_FORMAT
Defines the format string for %(asctime)s in log
records. Default: None . This option is ignored if
@ -165,10 +168,16 @@ optional arguments:
Template for report. Can be a file name or one of
aliases: "interactive", "json". Defaults to
"interactive".
--reuse-stack-name REUSE_STACK_NAME
Name of an existing Shaker heat stack to reuse. The
default is to not reuse an existing stack. Caution
should be taken to only reuse stacks meant for a
specific scenario. Also certain configs e.g. image-
name, flavor-name, stack-name, etc will be ignored
when reusing an existing stack.
--scenario SCENARIO Comma-separated list of scenarios to play. Each entity
can be a file name or one of aliases:
"misc/instance_metadata", "misc/static_agent",
"misc/static_agents_pair",
"misc/instance_metadata",
"openstack/cross_az/full_l2",
"openstack/cross_az/full_l3_east_west",
"openstack/cross_az/full_l3_north_south",
@ -208,6 +217,9 @@ optional arguments:
--server-endpoint SERVER_ENDPOINT
Address for server connections (host:port), defaults
to env[SHAKER_SERVER_ENDPOINT].
--stack-name STACK_NAME
Name of test heat stack. The default is a uniquely
generated name.
--subunit SUBUNIT Subunit stream file name, defaults to
env[SHAKER_SUBUNIT].
--syslog-log-facility SYSLOG_LOG_FACILITY

73
etc/shaker.conf
View File

@ -14,7 +14,7 @@
# files, see the Python logging module documentation. Note that when logging
# configuration files are used then all logging configuration is set in the
# configuration file and other logging configuration options are ignored (for
# example, logging_context_format_string). (string value)
# example, log-date-format). (string value)
# Note: This option can be changed without restarting.
# Deprecated group/name - [DEFAULT]/log_config
#log_config_append = <None>
@ -65,27 +65,63 @@
# set. (boolean value)
#use_stderr = false
# Format string to use for log messages with context. (string value)
# Log output to Windows Event Log. (boolean value)
#use_eventlog = false
# The amount of time before the log files are rotated. This option is ignored
# unless log_rotation_type is setto "interval". (integer value)
#log_rotate_interval = 1
# Rotation interval type. The time of the last file change (or the time when
# the service was started) is used when scheduling the next rotation. (string
# value)
# Possible values:
# Seconds - <No description provided>
# Minutes - <No description provided>
# Hours - <No description provided>
# Days - <No description provided>
# Weekday - <No description provided>
# Midnight - <No description provided>
#log_rotate_interval_type = days
# Maximum number of rotated log files. (integer value)
#max_logfile_count = 30
# Log file maximum size in MB. This option is ignored if "log_rotation_type" is
# not set to "size". (integer value)
#max_logfile_size_mb = 200
# Log rotation type. (string value)
# Possible values:
# interval - Rotate logs at predefined time intervals.
# size - Rotate logs once they reach a predefined size.
# none - Do not rotate log files.
#log_rotation_type = none
# Format string to use for log messages with context. Used by
# oslo_log.formatters.ContextFormatter (string value)
#logging_context_format_string = %(asctime)s.%(msecs)03d %(process)d %(levelname)s %(name)s [%(request_id)s %(user_identity)s] %(instance)s%(message)s
# Format string to use for log messages when context is undefined. (string
# value)
# Format string to use for log messages when context is undefined. Used by
# oslo_log.formatters.ContextFormatter (string value)
#logging_default_format_string = %(asctime)s.%(msecs)03d %(process)d %(levelname)s %(name)s [-] %(instance)s%(message)s
# Additional data to append to log message when logging level for the message
# is DEBUG. (string value)
# is DEBUG. Used by oslo_log.formatters.ContextFormatter (string value)
#logging_debug_format_suffix = %(funcName)s %(pathname)s:%(lineno)d
# Prefix each line of exception output with this format. (string value)
# Prefix each line of exception output with this format. Used by
# oslo_log.formatters.ContextFormatter (string value)
#logging_exception_prefix = %(asctime)s.%(msecs)03d %(process)d ERROR %(name)s %(instance)s
# Defines the format string for %(user_identity)s that is used in
# logging_context_format_string. (string value)
# logging_context_format_string. Used by oslo_log.formatters.ContextFormatter
# (string value)
#logging_user_identity_format = %(user)s %(tenant)s %(domain)s %(user_domain)s %(project_domain)s
# List of package logging levels in logger=LEVEL pairs. This option is ignored
# if log_config_append is set. (list value)
#default_log_levels = amqp=WARN,amqplib=WARN,boto=WARN,qpid=WARN,sqlalchemy=WARN,suds=INFO,oslo.messaging=INFO,oslo_messaging=INFO,iso8601=WARN,requests.packages.urllib3.connectionpool=WARN,urllib3.connectionpool=WARN,websocket=WARN,requests.packages.urllib3.util.retry=WARN,urllib3.util.retry=WARN,keystonemiddleware=WARN,routes.middleware=WARN,stevedore=WARN,taskflow=WARN,keystoneauth=WARN,oslo.cache=INFO,dogpile.core.dogpile=INFO
#default_log_levels = amqp=WARN,amqplib=WARN,boto=WARN,qpid=WARN,sqlalchemy=WARN,suds=INFO,oslo.messaging=INFO,oslo_messaging=INFO,iso8601=WARN,requests.packages.urllib3.connectionpool=WARN,urllib3.connectionpool=WARN,websocket=WARN,requests.packages.urllib3.util.retry=WARN,urllib3.util.retry=WARN,keystonemiddleware=WARN,routes.middleware=WARN,stevedore=WARN,taskflow=WARN,keystoneauth=WARN,oslo.cache=INFO,oslo_policy=INFO,dogpile.core.dogpile=INFO
# Enables or disables publication of error events. (boolean value)
#publish_errors = false
@ -136,6 +172,10 @@
# Authentication project domain name. Defaults to env[OS_PROJECT_DOMAIN_NAME].
# (string value)
#
# This option has a sample default set, which means that
# its actual default value may vary from the one documented
# below.
#os_project_domain_name = Default
# Authentication username, defaults to env[OS_USERNAME]. (string value)
@ -146,6 +186,10 @@
# Identity API version, defaults to env[OS_IDENTITY_API_VERSION]. (string
# value)
#
# This option has a sample default set, which means that
# its actual default value may vary from the one documented
# below.
#os_identity_api_version = 3
# Authentication password, defaults to env[OS_PASSWORD]. (string value)
@ -187,13 +231,22 @@
# value)
#flavor_name = shaker-flavor
# Name of test heat stack. The default is a uniquely generated name. (string
# value)
#stack_name = <None>
# Name of an existing Shaker heat stack to reuse. The default is to not reuse
# an existing stack. Caution should be taken to only reuse stacks meant for a
# specific scenario. Also certain configs e.g. image-name, flavor-name, stack-
# name, etc will be ignored when reusing an existing stack. (string value)
#reuse_stack_name = <None>
# Clean up the heat-stack upon any error occurred during scenario execution.
# (boolean value)
#cleanup_on_error = true
# Comma-separated list of scenarios to play. Each entity can be a file name or
# one of aliases: "misc/instance_metadata", "misc/static_agent",
# "misc/static_agents_pair", "openstack/cross_az/full_l2",
# one of aliases: "misc/instance_metadata", "openstack/cross_az/full_l2",
# "openstack/cross_az/full_l3_east_west",
# "openstack/cross_az/full_l3_north_south", "openstack/cross_az/perf_l2",
# "openstack/cross_az/perf_l3_east_west",

12
shaker/engine/config.py
View File

@ -162,6 +162,18 @@ OPENSTACK_OPTS = [
default=utils.env('SHAKER_FLAVOR') or 'shaker-flavor',
help='Name of image flavor. The default is created by '
'shaker-image-builder.'),
cfg.StrOpt('stack-name',
default=utils.env('SHAKER_STACK_NAME') or None,
help='Name of test heat stack. The default is a uniquely '
'generated name.'),
cfg.StrOpt('reuse-stack-name',
default=utils.env('SHAKER_REUSE_STACK_NAME') or None,
help='Name of an existing Shaker heat stack to reuse. The '
'default is to not reuse an existing stack. Caution '
'should be taken to only reuse stacks meant for a '
'specific scenario. Also certain configs e.g. '
'image-name, flavor-name, stack-name, etc will be '
'ignored when reusing an existing stack.'),
cfg.BoolOpt('cleanup-on-error',
default=(utils.env('SHAKER_CLEANUP_ON_ERROR') or True),
help='Clean up the heat-stack upon any error occurred during '

12
shaker/engine/deploy.py
View File

@ -263,7 +263,15 @@ class Deployment(object):
self.flavor_name = flavor_name
self.image_name = image_name
if cfg.CONF.stack_name is not None:
self.stack_name = cfg.CONF.stack_name
else:
self.stack_name = 'shaker_%s' % utils.random_string()
if cfg.CONF.reuse_stack_name is not None:
self.stack_name = cfg.CONF.reuse_stack_name
self.dns_nameservers = dns_nameservers
# intiailizing self.external_net last so that other attributes don't
# remain uninitialized in case user forgets to create external network
@ -333,9 +341,13 @@ class Deployment(object):
if support_templates is not None:
self._deploy_support_stacks(support_templates, base_dir)
if cfg.CONF.reuse_stack_name is None:
self.stack_id = heat.create_stack(
self.openstack_client.heat, self.stack_name, rendered_template,
merged_parameters, env_file)
else:
self.stack_id = heat.get_id_with_name(self.openstack_client.heat,
self.stack_name)
# get info about deployed objects
outputs = heat.get_stack_outputs(self.openstack_client.heat,

10
shaker/openstack/clients/heat.py
View File

@ -51,6 +51,16 @@ def get_stack_status(heat_client, stack_id):
raise exc.HTTPNotFound(message='Stack %s is not found' % stack_id)
def get_id_with_name(heat_client, stack_name):
# This method isn't really necessary since the Heat client accepts
# stack_id and stack_name interchangeably. This is provided more as a
# safety net to use ids which are guaranteed to be unique and provides
# the benefit of keeping the Shaker code consistent and more easily
# traceable.
stack = heat_client.stacks.get(stack_name)
return stack.id
def wait_stack_completion(heat_client, stack_id):
reason = None
status = None