rally-openstack/doc/source/verification/howto/migrate_from_old_design.rst

HowTo migrate from Verification component 0.7.0 to 0.8.0

Note

This document describes migration process from 0.7.0 to 0.8.0 Rally version. You can apply this instruction for migration to later versions, but check all references and release notes before trying to do it.

Verification Component was introduced long time ago even before the first Rally release. It started as a small helper thing but became a big powerful tool. Since it was not designed to all features that were implemented there later, it contained a lot of workarounds and hacks.

New Verification Component, which we are happy to introduce, should fix all architecture issues and improve user-experience. Unfortunately, fixing all those obsolete architecture decisions could not be done in a backward-compatible way, or it would produce much more workarounds. That is why we decided to redesign the whole component in a clear way - remove old code and write a new one from scratch.

Migration to New Verification Component should be simple and do not take too much time. You can find description of made changes below.

Reports

We completely reworked verification reports and merged comparison to main report. Now you can build one report for multiple number of verifications.

For more details follow verification-reports

Verification statuses

Old Status	New Status	Description
init	init	Initial state. It appears instantly after calling rally verify start command before the actual run of verifier's tool.
running		It was used right after checking status of verifier. It is redundant in terms of new design.
verifying	running	Identifies the process of execution tool.
finished	finished ------------+ failed	Previously, finished state was used for identification of just finished verification. By "finished" meant that verification has any test result. Now it means that verification was executed and doesn't have failures, unexpected success or any kind of errors. ---------------------------------------------------+ Old purpose is identification "errors", situations when results are empty. The right use is an identification finished verification with tests in failed and unexpected success statuses.
failed	crashed	Something went wrong while launching verification.

Latest information about verification statuses you can find at verification_statuses.

Command Line Interface

You can find the latest information about Verification Component CLI here -rally-verify-cli-reference.

Installing verifier

Command for Rally 0.7.0 - rally verify install

$ rally verify install --deployment <uuid> --source <url> --version <vers> \
  --system-wide

Command for Rally 0.8.0:

$ rally verify create-verifier --type "tempest" --source <url> \
  --version <version> --system-wide

Here you can find several important improvements:

1. Rally team introduced new entity - verifiers. Verifier stores all information about installed tool (i.e., source, version, system-wide) in a database. You do not need to transmit the same arguments into all rally verify commands as it was previously with --system-wide flag.
2. You can use particular verifier for multiple deployments. --deployment flag moved to rally verify start command. Also, you can run it simultaneously (checking in parallel different sets, different cloud, etc)
3. Verification Component can use not only Tempest for verifying system. Check known-verifier-types for full list of supported tools.
4. You can have unlimited number of verifiers.

Re-install verifier aka update

Command for Rally 0.7.0 - rally verify reinstall

$ rally verify reinstall --deployment <uuid> --source <url> --version <vers> \
  --system-wide

Command for Rally 0.8.0:

$ rally verify update-verifier --id <id> --source <url> --version <vers> \
  --system-wide --no-system-wide --update-venv

Changes:

1. rally verify update-verifier doesn't require deployment id
2. You can switch between usage of system-wide installation and virtual environment.
3. You can update just virtual environment without cloning verifier code again

Uninstall

Command for Rally 0.7.0 - rally verify uninstall

$ rally verify uninstall --deployment <uuid>

Command for Rally 0.8.0:

$ rally verify delete-verifier --id <id> --deployment-id <id> --force

Changes:

1. As it was mentioned before, Verifier doesn't have an alignment to any particular deployment, so deployment argument is optional now. If --deployment-id argument is specified only deployment specific data will be removed (i.e, configurations).
2. New --force flag for removing all verifications results for that verifier.

Installation extensions

Command for Rally 0.7.0 - rally verify installplugin

$ rally verify installplugin --deployment <uuid> --source <url> \
  --version <vers> --system-wide

Command for Rally 0.8.0:

$ rally verify add-verifier-ext --id <id> --source <url> --version <vers> \
  --extra-settings <data>

Changes:

1. --system-wide flag is removed. Rally checks the verifier information to identify where to install the extension - in a system-side way or use virtual environment.
2. New --extra-settings flag. In case of Tempest, it is redundant, but for other verifiers allows to transmit some extra installation settings for verifier extension.

Uninstall extensions

Command for Rally 0.7.0 - rally verify uninstallplugin

$ rally verify uninstallplugin --deployment <uuid> --repo-name <repo_name> \
  --system-wide

Command for Rally 0.8.0:

$ rally verify delete-verifier-ext --id <id> --name <name>

Changes:

1. It is one more place where you do not need to pass --system-wide flag anymore.
2. --deployment flag is gone.
3. --repo-name is renamed to just --name.

List extensions

Command for Rally 0.7.0 - rally verify listplugins

$ rally verify listplugins --deployment <uuid> --system-wide

Command for Rally 0.8.0:

$ rally verify list-verifier-exts --id <id>

Changes:

1. No need to specify --system-wide flag.
2. --deployment flag is gone.

Discover available tests

Command for Rally 0.7.0 - rally verify discover

$ rally verify discover --deployment <uuid> --system-wide --pattern <pattern>

Command for Rally 0.8.0:

$ rally verify list-verifier-tests --id <id> --pattern <pattern>

Changes:

1. No need to specify --system-wide flag.
2. --deployment flag is gone.

Configuring

Commands for Rally 0.7.0:

• The command for generating configs rally verify genconfig

  $ rally verify genconfig --deployment <uuid> --tempest-config <path> \
    --add-options <path> --override

• The command for showing configs rally verify showconfig

  $ rally verify showconfig --deployment <uuid>

Command for Rally 0.8.0:

$ rally verify configure-verifier --id <id> --deployment-id <uuid> \
  --extend <path/json/yaml> --override <path> --reconfigure --show

Changes:

1. The argument --override replaces old --tempest-config name. First of all, argument name "override" is a unified word without alignment to any tool. Also, it describes in the best way the meaning of the action: use client specified configuration file.

2. The argument --extend replaces old --add-options. It accepts a path to config in INI format or JSON/YAML string. In future, it will be extended with the ability to specify a path to JSON/YAML file.

3. The argument --reconfigure replaces old --override. It means that existing file will be ignored and new one will be used/created.

4. If the argument --show is specified, a configuration of verifier will be displayed at the end of command execution.

   Note

   We do not have a separate command for showing configurations anymore. rally verify configure-verifier --show shows an existing configuration if it exists and --reconfigure argument is not specified

Show config

Command for Rally 0.7.0 - rally verify showconfig

$ rally verify showconfig --deployment <uuid>

Command for Rally 0.8.0:

$ rally verify configure-verifier --id <id> --deployment-id <uuid> --show

Changes:

We do not have a separate command for that task. rally verify configure-verifier --show shows an existing configuration (if it exists) in case of not specified --reconfigure argument.

Running verification

Command for Rally 0.7.0 - rally verify start

$ rally verify start --deployment <uuid> --set <set_name> --regex <regex> \
  --load-list <path> --tests-file <path> --skip-list <path> \
  --tempest-config <path> --xfail-list <path> --system-wide \
  --concurrency <N> --failing --no-use

Command for Rally 0.8.0:

$ rally verify start --id <id> --deployment-id <uuid> --pattern <pattern> \
  --load-list <path> --skip-list <path> --xfail-list <path> \
  --concurrency <N> --no-use

Changes:

1. You need to pass verifier id
2. Arguments --set and --regex are merged in the new model to single --pattern argument. Name of tests set should be specified like --pattern set_name=<set_name>. It was done to provide a way for each verifier to support custom arguments.
3. The argument --tests-file was deprecated in Rally 0.6.0 and time&ability come to delete it.
4. Arguments --skip-list and --xfail-list accept path to file in JSON/YAML format. Content should be a dictionary, where keys are tests names (full name with id and tags) and values are reasons.
5. The argument --tempest-config is gone. Use rally verify configure-verifier --id <id> --deployment-id <uuid> --override <path> instead.
6. The argument --system-wide is gone like in most of other commands.

Show verification result

Commands for Rally 0.7.0:

• The command for showing results of verification rally verify show

  $ rally verify show --uuid <uuid> --sort-by <query> --detailed

• Separate command witch call rally verify show with hardcoded --detailed flag rally verify showconfig

  $ rally verify detailed --uuid <uuid> --sort-by <query>

Command for Rally 0.8.0:

$ rally verify show --uuid <uuid> --sort-by <query> --detailed

Changes:

1. Redundant rally verify detailed command is removed
2. Sorting tests via --sort-by argument is extended to name/duration/status

Listing all verifications

Command for Rally 0.7.0 - rally verify list

$ rally verify list

Command for Rally 0.8.0:

$ rally verify list --id <id> --deployment-id <id> --status <status>

Changes:

You can filter verifications by verifiers, by deployments and by results statuses.

Importing results

Command for Rally 0.7.0 - rally verify import

$ rally verify import --deployment <uuid> --set <set_name> --file <path> --no-use

Command for Rally 0.8.0:

$ rally verify import --id <id> --deployment <uuid> --file <path> \
  --run-args <run_args> --no-use

Changes:

1. You need to specify verifier to import results for.
2. The argument --set is merged into unified --run-args.

Building reports

Commands for Rally 0.7.0:

• The command for building HTML/JSON reports of verification rally verify results

  $ rally verify results --uuid <uuid> --html --json --output-file <path>

• The command for comparison two verifications rally verify compare

  $ rally verify compare --uuid-1 <uuid_1> --uuid-2 <uuid_2> --csv --html \
    --json --output-file <output_file> --threshold <threshold>

Command for Rally 0.8.0:

$ rally verify report --uuid <uuid> --type <type> --to <destination> --open

Changes:

1. Building reports becomes pluggable. You can extend reporters types. See verification-reports for more details.
2. The argument --type expects type of report (HTML/JSON). There are no more separate arguments for each report type.
3. Reports are not aligned to only local types, so the argument --to replaces --output-file. In case of HTML/JSON reports, it can include a path to the local file like it was previously or URL to some external system with credentials like https://username:password@example.com:777.
4. The comparison is embedded into main reports and it is now limited by two verifications results. There are no reasons for the separate command for that task.

The End

Have nice verifications!