airshipctl/docs/source/testing-guidelines.md

Testing Guidelines

This document lays out several guidelines to ensure quality and consistency throughout airshipctl's test bed.

Testing packages

The airshipctl project uses the testify library, a thin wrapper around Go's builtin testing package. The testify package provides the following packages:

• assert: Functions from this package can be used to replace most calls to t.Error
• require: Contains the same functions as above, but these functions should replace calls to t.Fatal
• mock: Contains the Mock mechanism, granting the ability to mock out structs

Test coverage

Tests should cover at least 80% of the codebase. Anything less will cause the CI gates to fail. A developer should assert that their code meets this criteria before submitting a change. This check can be performed with one of the following make targets:

# Runs all unit tests, then computes and reports the coverage
make cover

Good practice is to assert that the changed packages have not decreased in coverage. The coverage check can be run for a specific package with a command such as the following.

make cover PKG=./pkg/foo

Additional testing should be done to ensure that the proposed change meets an expected level of quality. These tests include:

# Tidy, to ensure go.mod is up to date
make tidy

# Lint, to ensure code meets linting requirements
make lint

# Update-golden, to ensure the golden test data reflects the current test cases
make update-golden

When the above are done, if you would like to perform the same dockerized container testing as the CI gates you can do so via:

make docker-image-test-suite

NOTE: If test cases are deleted you must first run make update-golden, and commit your changes prior to running the docker-image-test-suite make target. Otherwise the docker-image-test-suite make target (and the CI job) will fail.

Test directory structure

Test files end in _test.go, and sit next to the tested file. For example, airshipctl/pkg/foo/foo.go should be tested by airshipctl/pkg/foo/foo_test.go. A test's package name should also end in _test, unless that file intends to test unexported fields and method, at which point it should be in the package under test.

Go will ignore any files stored in a directory called testdata, therefore all non-Go test files (such as expected output or example input) should be stored there.

Any mocks for a package should be stored in a sub-package ending in mocks. Each mocked struct should have its own file, where the filename describes the struct, i.e. a file containing a mocked Fooer should be stored at mocks/fooer.go. Mocked files can be either handwritten or generated via mockery. The mockery tool can generate files in this fashion with the following command.

mockery -all -case snake

An example file structure might look something like the following.

airshipctl/pkg/foo
├── foo.go
├── foo_test.go
├── mocks
│   └── fooer.go
└── testdata
    └── example-input.yaml

Testing guidelines

This section annotates various standards for unit tests in airshipctl. These should be thought of as "guidelines" rather than "rules".

• Using table-tests prevents a lot of duplicated code.
• Using subtests allows tests to provide much more fine-grained results.
• Calls to methods from testify/require be reserved for situations in which the test should fail immediately (e.g. during test setup). Generally, testify/assert should be preferred.

How to write unit tests for files listed under the cmd package

Go files listed under the cmd package should be relatively slim. Their purpose is to be a client of the pkg package. Most of these files will contain no more than a single function which creates and returns a cobra.Command. Nonetheless, these functions need to be tested. To help alleviate some of the difficulties that come with testing a CLI, airshipctl provides several helper structs and functions under the testutil package.

As an example, suppose you have the following function:

func NewVersionCommand() *cobra.Command {
	versionCmd := &cobra.Command{
		Use:   "version",
		Short: "Show the version number of airshipctl",
		Run: func(cmd *cobra.Command, args []string) {
			out := cmd.OutOrStdout()
			clientV := version.clientVersion()
			w := util.NewTabWriter(out)
			defer w.Flush()
			fmt.Fprintf(w, "%s:\t%s\n", "airshipctl", clientV)
		},
	}
	return versionCmd
}

Testing this functionality is easy with the use of the pre-built testutil.CmdTest:

func TestVersion(t *testing.T) {
	versionCmd := cmd.NewVersionCommand()
	cmdTests := []*testutil.CmdTest{
		{
			Name:    "version",
			CmdLine: "",
			Cmd:     versionCmd,
			Error:   nil,
		},
		{
			Name:    "version-help",
			CmdLine: "--help",
			Cmd:     versionCmd,
			Error:   nil,
		},
	}

	for _, tt := range cmdTests {
		testutil.RunTest(t, tt)
	}
}

The above test uses CmdTest structs, which are then fed to the RunTest function. This function provides abstraction around running a command on the command line and comparing its output to a "golden file" (the pre-determined expected output). The following describes the fields of the CmdTest struct.

• Name - The name for this test. This field must be unique, as it will be used while naming the golden file
• CmdLine - The arguments and flags to pass to the command
• Cmd - The actual instance of a cobra.Command to run. The above example reuses the command, but more complex tests may require different instances (e.g. to pass in a different Settings object)
• Error - The expected error for the command to return. This can be omitted if this test doesn't expect an error

Once you've written your test, you can generate the associated golden files by running make update-golden, which invokes the "update" mode for testutil.RunTest. When the command has completed, you can view the output in the associated files in the testdata directory next to your command. Note that these files are easily discoverable from the output of git status. When you're certain that the golden files are correct, you can add them to the repo.