Testing the infrastructure means, in short, performing a system validation, i.e. determining whether a particular server or software works in the desired way.
It is carried out at different levels (including through deployment of monitoring and alarming systems) and in different phases of an application’s life cycle – for example, before installation, while the application is running or during installation. The most important question here is: “On what basis are we able to conclude that the installation was successful?”. The simplest solution seems to be to log on to the machine and check that the desired services or containers are running. The problem starts when we do not have 1 machine but 100; logging in and verifying each of them would be a time-consuming and tedious task – this creates the risk of errors – have we really verified the operation of all the necessary elements? In addition, the requirements expand as the project grows and the list of things to check grows longer. It is not difficult to imagine a situation where validation has not been performed correctly on one of the numerous machines. How to simplify the process of performing this type of validation? The answer is the TestInfra framework.
TestInfra – the framework for testing your infrastructure
Writing unit tests that check the infrastructure is a broad topic. The Testinfra Framework is something that can make this task easier. It is an open solution integrated with the python testing framework Pytest. This framework allows for creating tests to validate the current state of designated servers that have previously been configured – either manually or using automated deployment and configuration tools such as Ansible, Salt, Puppet or Chef. In theory, these tools verify the status and operation of installed services or packages, however, in some cases the configuration has not been done in the right way – something may have stopped working right after installation or the configuration files have not been updated with a newly needed dependency. Therefore, it is particularly important to perform the additional independent validation of the correct installation and configuration of the indicated server. In modern software deployment, most of the work is done automatically using automation servers such as Jenkins or TeamCity, which perform the configuration of the designated machine, so it seems appropriate to add an additional step: running tests in the configuration pipeline for a given machine. When automating this type of testing, it can be beneficial to generate a list of hosts along with an indication on how they establish connections. TestInfra offers several types of connections to machines, giving you the possibility to tailor the tests to your needs. TestInfra has built-in integration with:
- localhost,
- Paramiko (Python implementation of SSH),
- Docker,
- SSH,
- Salt,
- Ansible,
- Kubernetes (via kubectl),
- WinRM.
The TestInfra Framework works with the python testing framework (Pytest) and uses so-called fixtures – helper functions injected into the tests. TestInfra’s main fixture is the host – a container that aggregates all the other attributes provided by the tool. The most commonly used elements of this fixture are as follows:
- host.ansible – provides full access to all Ansible properties while the test is running, for example the hosts, inventory and vars properties,
- host.addr – a network tool to check IPv4 and IPv6 addresses – whether the host is available and resolve the name,
- host.docker – a proxy for the Docker API – allows you to interact with containers and see if they work,
- host.interface – a helper tool to check addresses from a specific interface,
- host.iptables – a tool used to verify firewall rules,
- host.mount_point – checks mounts and file system types based on paths and mount options,
- host.package – asks if the specified package is installed, and if so, what version,
- host.process – checks the running processes,
- host.socket – tests listening on tcp/udp ports,
- host.system_info – provides various types of system metadata such as distribution version, release, and code name,
- host.check_output – runs a system command, checks its output, and verifies that the returned command code is different from 0,
- host.run – runs the command, allows to check the returned value and the contents of the host.stderr and host.stdout streams,
- host.run_expect – checks if the value returned by the command is as expected.
Test your infrastructure in Python
You will need a computer with Python (version 3.6 or more recent) installed to run the tests. The following examples were implemented using Python 3.8 and the Ubuntu operating system (20.04 LTE).
You will need a new virtual environment with Pytest to get started:
$ python3 -m venv validation
$ source validation/bin/activate
$(validation) pip install pytest
After installing Pytest, install the TestInfra package:
$(validation) pip install testinfra
Environment prepared, time to write the first test. We create a file with the first test:
def test_system_info(host):
assert host.system_info.type == “linux”, “Host should be running on linux”
assert host.system_info.distribution == “centos”, “Host should be running on centos”
assert host.system_info.release == “8”, “Centos should be in 8 version”
A major advantage of TestInfra is its readability – even without knowledge of the API you can see what the test does – it checks system information – whether Linux is installed, whether the Linux distribution is Centos, and whether the Centos installed is version 8. Such a test can be run from the console using the command:
$(validation) pytest –hosts=’ssh://{adres_hosta_do_sprawdzenia}’ test.py
One element that is commonly checked is the correctness of the installed packages on the remote machine. An example of a test to check if the nginx package is installed with version 1.6 looks as follows:
def test_package_nginx_is_installed_in_version_1_6(host):
package = host.package(“nginx”)
assert package.is_installed, “Nginx should be installed”
assert package.version.startswith(“1.6”), “Nginx should be installed in version 1.6”
If the test outcome is positive, a message will be displayed:
If, on the other hand, the nginx package has not been installed, an appropriate message will be displayed:
Most often, however, we will be not be testing individual packages, but an entire list. This is when the ability of the Pytest framework to parameterise tests comes in handy. By using the @pytest.mark.parametrize decorator, we are able to very quickly extend the test for checking installed packages with additional elements:
import pytest
@pytest.mark.parametrize(“package_name, package_version”, [(“nginx”, “1.6”), (“python3”, “3”), (“docker”, “1.6”)])
def test_package__is_installed_in_requested_version(host, package_name, package_version):
package = host.package(package_name)
assert package.is_installed, f”Package {package_name} should be installed”
assert package.version.startswith(package_version), f”Package {package_name} should be installed in version {package_version}”
Three tests were quickly created from one test; these can be easily extended with further elements. After running such a set, we get information on which packages are installed and which are not:
As you can see from the framework messages, the nginx package is not installed at all, the python3 package is installed in the correct version, and the docker package is installed, but its version (1.5.2) does not match the version that was required (1.6).
The above examples are only a very small part of the modules provided by the TestInfra Framework; the full list of modules along with usage examples can be found at https://testinfra.readthedocs.io/en/latest/modules.html.
