pytest-easy-server - Pytest plugin for easy testing against servers

The pytest-easy-server package is a Pytest plugin that provides a Pytest fixture fixture es_server() that resolves to the set of servers the tests should run against.

The set of servers is defined in a server file (aka “easy-server file”) and the secrets to access the servers are defined in a vault file that is referenced by the server file, in the formats defined by the easy-server package.

The files to use and the server or group nickname to select for the test can be specified in pytest options added by the plugin:

--es-file=FILE
                        Path name of the easy-server file to be used.
                        Default: es_server.yml in current directory.
--es-nickname=NICKNAME
                        Nickname of the server or server group to test against.
                        Default: The default from the server file.

Usage

Supported environments

The pytest-easy-server package is supported in these environments:

• Operating Systems: Linux, macOS / OS-X, native Windows, Linux subsystem in Windows, UNIX-like environments in Windows.

• Python: 2.7, 3.4, and higher

Installation

The following command installs the pytest-easy-server package and its prerequisite packages into the active Python environment:

$ pip install pytest-easy-server

When Pytest runs, it will automatically find the plugin and will show its version, e.g.:

plugins: easy-server-0.5.0

Server file and vault file

The server file define the servers, server groups and a default server or group. It is described in the “easy-server” documentation in section Server files.

The vault file defines the secrets needed to access the servers and can stay encrypted in the file system while being used. It is described in the “easy-server” documentation in section Vault files.

The servers and groups are identified with user-defined nicknames.

Using the es_server fixture

If your pytest test function uses the es_server() fixture, the test function will be invoked for the server specified in the --es-nickname command line option, or the set of servers if the specified nickname is that of a server group.

From a perspective of the test function that is invoked, the fixture resolves to a single server item.

The following example shows a test function using this fixture and how it gets to the details for accessing the server:

from pytest_easy_server import es_server

def test_sample(es_server):
    """
    Example Pytest test function that tests something.

    Parameters:
      es_server (easy_server.Server): Pytest fixture; the server to be
        used for the test
    """

    # Standard properties from the server file:
    nickname = es_server.nickname
    description = es_server.description

    # User-defined additional properties from the server file:
    stuff = es_server.user_defined['stuff']

    # User-defined secrets from the vault file:
    host = es_server.secrets['host']
    username = es_server.secrets['username']
    password = es_server.secrets['password']

    # Session to server using a fictitious session class
    session = MySession(host, username, password)

    # Test something
    result = my_session.perform_function()
    assert ...

    # Cleanup
    session.close()

The example shows how to access the standard and user-defined properties from the “easy-server” file for demonstration purposes. The data structure of the user-defined properties in the server file and of the secrets in the vault file is completely up to you, so you could decide to have the host and userid in user-defined properties in the server file, and have only the password in the vault file.

The es_server parameter of the test function is a easy_server.Server object that represents a server item from the server file for testing against a single server. It includes the corresponding secrets item from the vault file.

Example pytest runs

The user-defined properties and vault secrets used in the test function shown in the previous section are from the examples located in the examples directory of the GitHub repo of the project. The pytest runs are performed from within that directory.

Example file es_server.yml:

vault_file: vault.yml

servers:

  myserver1:                            # Nickname of the server
    description: "my dev system 1"
    contact_name: "John Doe"
    access_via: "VPN to dev network"
    user_defined:                       # User-defined additional properties
      stuff: "more stuff"

  myserver2:
    description: "my dev system 2"
    contact_name: "John Doe"
    access_via: "intranet"
    user_defined:
      stuff: "more stuff"

server_groups:

  mygroup1:
    description: "my dev systems"
    members:
      - myserver1
      - myserver2

default: mygroup1

Example file es_vault.yml:

secrets:

  myserver1:
    host: "10.11.12.13"                 # User-defined properties
    username: myuser1
    password: mypass1

  myserver2:
    host: "9.10.11.12"                  # User-defined properties
    username: myuser2
    password: mypass2

The directory also contains a test module test_function.py with a test function that uses the es_server() fixture and prints the attributes and secrets for the server it is invoked for.

The following pytest runs use the default server file (es_server.yml in the current directory) and the vault password had been prompted for already and is now found in the keyring service.

In pytest verbose mode, the pytest-easy-server plugin prints messages about which server file is used and where the vault password comes from.

In this pytest run, the default nickname is used (the default defined in the server file, which is group mygroup1 that contains servers myserver1 and myserver2):

$ pytest -s -v .
==================================================== test session starts ====================================================
platform darwin -- Python 3.8.7, pytest-6.2.2, py-1.10.0, pluggy-0.13.1 -- /Users/maiera/virtualenvs/pytest-es38/bin/python
cachedir: .pytest_cache
rootdir: /Users/maiera/PycharmProjects/pytest-easy-server
plugins: easy-server-0.5.0.dev1
collecting ...
pytest-easy-server: Using server file /Users/maiera/PycharmProjects/pytest-easy-server/examples/es_server.yml
pytest-easy-server: Using vault password from prompt or keyring service.
collected 2 items

test_function.py::test_sample[es_server=myserver1]
MySession: host=10.11.12.13, username=myuser1, password=mypass1
PASSED
test_function.py::test_sample[es_server=myserver2]
MySession: host=9.10.11.12, username=myuser2, password=mypass2
PASSED

===================================================== 2 passed in 0.02s =====================================================

In this pytest run, the nickname is specified in the pytest command line using the --es-nickname option, to run only on server myserver1:

$ pytest -s -v . --es-nickname myserver1
==================================================== test session starts ====================================================
platform darwin -- Python 3.8.7, pytest-6.2.2, py-1.10.0, pluggy-0.13.1 -- /Users/maiera/virtualenvs/pytest-es38/bin/python
cachedir: .pytest_cache
rootdir: /Users/maiera/PycharmProjects/pytest-easy-server
plugins: easy-server-0.5.0.dev1
collecting ...
pytest-easy-server: Using server file /Users/maiera/PycharmProjects/pytest-easy-server/examples/es_server.yml
pytest-easy-server: Using vault password from prompt or keyring service.
collected 1 item

test_function.py::test_sample[es_server=myserver1]
MySession: host=10.11.12.13, username=myuser1, password=mypass1
PASSED

===================================================== 1 passed in 0.02s =====================================================

Controlling which servers to test against

When pytest loads the pytest-easy-server plugin, its set of command line options gets extended by those contributed by the plugin. These options allow controlling which server file is used and wich server or server group is used to test against. These options are optional and have sensible defaults:

--es-file=FILE
                        Path name of the easy-server file to be used.
                        Default: es_server.yml in current directory.
--es-nickname=NICKNAME
                        Nickname of the server or server group to test against.
                        Default: The default from the server file.

Running pytest as a developer

When running pytest with the pytest-easy-server plugin on your local system, you are prompted (in the command line) for the vault password upon first use of a particular vault file. The password is then stored in the keyring service of your local system to avoid future such prompts.

The section Running pytest in a CI/CD system describes the use of an environment variable to store the password which avoids the password prompt. For security reasons, you should not use this approach when you run pytest locally. The one password prompt can be afforded, and subsequent retrieval of the vault password from the keyring service avoids further prompts.

Running pytest in a CI/CD system

When running pytest with the pytest-easy-server plugin in a CI/CD system, you must set an environment variable named “ES_VAULT_PASSWORD” to the vault password.

This can be done in a secure way by defining a corresponding secret in the CI/CD system in your test run configuration (e.g. GitHub Actions workflow), and by setting that environment variable to the CI/CD system secret.

Here is a snippet from a GitHub Actions workflow that does that, using a secret that is also named “ES_VAULT_PASSWORD”:

- name: Run test
  env:
    ES_VAULT_PASSWORD: ${{ secrets.ES_VAULT_PASSWORD }}
  run: |
    pytest -s -v test_dir

The pytest-easy-server plugin picks up the vault password from the “ES_VAULT_PASSWORD” environment variable if set and the presence of that variable causes it not to prompt for the password and also not to store it in the keyring service (which would be useless since it is on the system that is used for the test run in the CI/CD system, and that is typically a new one for every run).

Security aspects

There are two kinds of secrets involved:

• The secrets in the vault file.

• The vault password.

The secrets in the vault file are protected if the vault file is encrypted in the file system. The functionality also works if the vault file is not encrypted, but the normal case should be that you keep it encrypted. If you store the vault file in a repository, make sure it is encrypted.

The vault password is protected in the following ways:

• When running pytest with the pytest-easy-server plugin on your local system, there is no permanent storage of the vault password anywhere except in the keyring service of your local system. There are no commands that take the vault password in their command line. The way the password gets specified is only in a password prompt, and then it is immediately stored in the keyring service and in future pytest runs retrieved from there.

  Your Python test programs of course can get at the secrets from the vault file (that is the whole idea after all). They can also get at the vault password by using the keyring service access functions but they have no need to deal with the vault password. In any case, make sure that the test functions do not print or log the vault secrets (or the vault password).

• When running pytest in a CI/CD system, the “ES_VAULT_PASSWORD” environment variable that needs to be set can be set from a secret stored in the CI/CD system. State of the art CI/CD systems go a long way of ensuring that these secrets cannot simply be echoed or otherwise revealed.

The keyring service provides access to the secrets stored there, for the user that is authorized to access it. The details on how this is done depend on the particular keyring service used and its configuration. For example, on macOS the keyring service periodically requires re-authentication.

Derived Pytest fixtures

If using the es_server() fixture in your test functions repeats boiler plate code for opening a session with the server, this can be put into a derived fixture.

The following fixture is an example for that. It opens and closes a session with a server using a fictitious class MySession:

In a file session_fixture.py:

import pytest
from pytest_easy_server import es_server

@pytest.fixture(scope='module')
def my_session(request, es_server):
    """
    Pytest fixture representing the set of MySession objects to use for
    testing against a server.
    """
    # Session to server using a fictitious session class
    session = MySession(
        host=es_server.secrets['host']
        username=es_server.secrets['username']
        password=es_server.secrets['password']
    )

    yield session

    # Cleanup
    session.close()

In your test functions, you can now use that fixture:

from pytest_easy_server import es_server  # Must still be imported
from session_fixture import my_session

def test_sample(my_session):
    """
    Example Pytest test function that tests something.

    Parameters:
      my_session (MySession): Pytest fixture; the session to the server
        to be used for the test
    """
    result = my_session.perform_function()  # Test something

A side note: Pylint and Flake8 do not recognize that ‘es_server’ and ‘my_session’ are fixtures that are interpreted by Pytest and thus complain about the unused ‘es_server’ and ‘my_session’ names, and about the ‘my_session’ parameter that redefines the global name. The following markup silences these tools:

# pylint: disable=unused-import
from pytest_easy_server import es_server  # noqa: F401
from session_fixture import my_session  # noqa: F401

def test_sample(my_session):  # pylint: disable=redefined-outer-name

API Reference

This section describes the API of the pytest-easy-server package. The API is kept stable using the compatibility rules defined for semantic versioning. An exception to this rule are fixes for security issues.

Any functions not described in this section are considered internal and may change incompatibly without warning.

es_server fixture

pytest_easy_server.es_server(request)

Pytest fixture representing a server item from an ‘easy-server’ file as a easy_server.Server object.

Pytest invokes testcases using this fixture for all servers to test against.

The servers are defined in a Server file and vault file. The servers to test against are controlled with pytest command line options as described in Controlling which servers to test against.

Returns

Server item for each server to test against.

Return type

easy_server.Server

Package version

pytest_easy_server.__version__ = '0.6.0'

The full version of this package including any development levels, as a string.

Possible formats for this version string are:

• “M.N.P.dev1”: Development level 1 of a not yet released version M.N.P

• “M.N.P”: A released version M.N.P

Development

This section only needs to be read by developers of the pytest-easy-server project, including people who want to make a fix or want to test the project.

Repository

The repository for the pytest-easy-server project is on GitHub:

https://github.com/andy-maier/pytest-easy-server

Setting up the development environment

1. If you have write access to the Git repo of this project, clone it using its SSH link, and switch to its working directory:

   $ git clone git@github.com:andy-maier/pytest-easy-server.git
   $ cd pytest-easy-server
   

   If you do not have write access, create a fork on GitHub and clone the fork in the way shown above.

2. It is recommended that you set up a virtual Python environment. Have the virtual Python environment active for all remaining steps.

3. Install the project for development. This will install Python packages into the active Python environment, and OS-level packages:

   $ make develop
   

4. This project uses Make to do things in the currently active Python environment. The command:

   $ make
   

   displays a list of valid Make targets and a short description of what each target does.

Building the documentation

The ReadTheDocs (RTD) site is used to publish the documentation for the project package at https://pytest-easy-server.readthedocs.io/

This page is automatically updated whenever the Git repo for this package changes the branch from which this documentation is built.

In order to build the documentation locally from the Git work directory, execute:

$ make builddoc

The top-level document to open with a web browser will be build_doc/html/docs/index.html.

Testing

All of the following make commands run the tests in the currently active Python environment. Depending on how the pytest-easy-server package is installed in that Python environment, either the directories in the main repository directory are used, or the installed package. The test case files and any utility functions they use are always used from the tests directory in the main repository directory.

The tests directory has the following subdirectory structure:

tests
 +-- unittest            Unit tests

There are multiple types of tests:

1. Unit tests

   These tests can be run standalone, and the tests validate their results automatically.

   They are run by executing:

   $ make test
   

   Test execution can be modified by a number of environment variables, as documented in the make help (execute make help).

   An alternative that does not depend on the makefile and thus can be executed from the source distribution archive, is:

   $ ./setup.py test
   

   Options for pytest can be passed using the --pytest-options option.

Contributing

Third party contributions to this project are welcome!

In order to contribute, create a Git pull request, considering this:

• Test is required.

• Each commit should only contain one “logical” change.

• A “logical” change should be put into one commit, and not split over multiple commits.

• Large new features should be split into stages.

• The commit message should not only summarize what you have done, but explain why the change is useful.

What comprises a “logical” change is subject to sound judgement. Sometimes, it makes sense to produce a set of commits for a feature (even if not large). For example, a first commit may introduce a (presumably) compatible API change without exploitation of that feature. With only this commit applied, it should be demonstrable that everything is still working as before. The next commit may be the exploitation of the feature in other components.

For further discussion of good and bad practices regarding commits, see:

• OpenStack Git Commit Good Practice

• How to Get Your Change Into the Linux Kernel

Further rules:

• The following long-lived branches exist and should be used as targets for pull requests:

  • master - for next functional version

  • stable_$MN - for fix stream of released version M.N.

• We use topic branches for everything!

  • Based upon the intended long-lived branch, if no dependencies

  • Based upon an earlier topic branch, in case of dependencies

  • It is valid to rebase topic branches and force-push them.

• We use pull requests to review the branches.

  • Use the correct long-lived branch (e.g. master or stable_0.2) as a merge target.

  • Review happens as comments on the pull requests.

  • At least one approval is required for merging.

• GitHub meanwhile offers different ways to merge pull requests. We merge pull requests by rebasing the commit from the pull request.

Releasing a version to PyPI

This section describes how to release a version of pytest-easy-server to PyPI.

It covers all variants of versions that can be released:

• Releasing a new major version (Mnew.0.0) based on the master branch

• Releasing a new minor version (M.Nnew.0) based on the master branch

• Releasing a new update version (M.N.Unew) based on the stable branch of its minor version

The description assumes that the andy-maier/pytest-easy-server Github repo is cloned locally and its upstream repo is assumed to have the Git remote name origin.

Any commands in the following steps are executed in the main directory of your local clone of the andy-maier/pytest-easy-server Git repo.

1. Set shell variables for the version that is being released and the branch it is based on:

   • MNU - Full version M.N.U that is being released

   • MN - Major and minor version M.N of that full version

   • BRANCH - Name of the branch the version that is being released is based on

   When releasing a new major version (e.g. 1.0.0) based on the master branch:

   MNU=1.0.0
   MN=1.0
   BRANCH=master
   

   When releasing a new minor version (e.g. 0.9.0) based on the master branch:

   MNU=0.9.0
   MN=0.9
   BRANCH=master
   

   When releasing a new update version (e.g. 0.8.1) based on the stable branch of its minor version:

   MNU=0.8.1
   MN=0.8
   BRANCH=stable_${MN}
   

2. Create a topic branch for the version that is being released:

   git checkout ${BRANCH}
   git pull
   git checkout -b release_${MNU}
   

3. Edit the version file:

   vi pytest_easy_server/_version.py
   

   and set the __version__ variable to the version that is being released:

   __version__ = 'M.N.U'
   

4. Edit the change log:

   vi docs/changes.rst
   

   and make the following changes in the section of the version that is being released:

   • Finalize the version.

   • Change the release date to today’s date.

   • Make sure that all changes are described.

   • Make sure the items shown in the change log are relevant for and understandable by users.

   • In the “Known issues” list item, remove the link to the issue tracker and add text for any known issues you want users to know about.

   • Remove all empty list items.

5. When releasing based on the master branch, edit the GitHub workflow file test.yml:

   vi .github/workflows/test.yml
   

   and in the on section, increase the version of the stable_* branch to the new stable branch stable_M.N created earlier:

   on:
     schedule:
       . . .
     push:
       branches: [ master, stable_M.N ]
     pull_request:
       branches: [ master, stable_M.N ]
   

6. Commit your changes and push the topic branch to the remote repo:

   git status  # Double check the changed files
   git commit -asm "Release ${MNU}"
   git push --set-upstream origin release_${MNU}
   

7. On GitHub, create a Pull Request for branch release_M.N.U. This will trigger the CI runs.

   Important: When creating Pull Requests, GitHub by default targets the master branch. When releasing based on a stable branch, you need to change the target branch of the Pull Request to stable_M.N.

8. On GitHub, close milestone M.N.U.

9. On GitHub, once the checks for the Pull Request for branch start_M.N.U have succeeded, merge the Pull Request (no review is needed). This automatically deletes the branch on GitHub.

10. Add a new tag for the version that is being released and push it to the remote repo. Clean up the local repo:

    git checkout ${BRANCH}
    git pull
    git tag -f ${MNU}
    git push -f --tags
    git branch -d release_${MNU}
    

11. When releasing based on the master branch, create and push a new stable branch for the same minor version:

    git checkout -b stable_${MN}
    git push --set-upstream origin stable_${MN}
    git checkout ${BRANCH}
    

    Note that no GitHub Pull Request is created for any stable_* branch.

12. On GitHub, edit the new tag M.N.U, and create a release description on it. This will cause it to appear in the Release tab.

    You can see the tags in GitHub via Code -> Releases -> Tags.

13. On ReadTheDocs, activate the new version M.N.U:

    • Go to https://readthedocs.org/projects/pytest-easy-server/versions/ and log in.

    • Activate the new version M.N.U.

      This triggers a build of that version. Verify that the build succeeds and that new version is shown in the version selection popup at https://pytest-easy-server.readthedocs.io/

14. Upload the package to PyPI:

    make upload
    

    This will show the package version and will ask for confirmation.

    Attention! This only works once for each version. You cannot release the same version twice to PyPI.

    Verify that the released version arrived on PyPI at https://pypi.python.org/pypi/pytest-easy-server/

Starting a new version

This section shows the steps for starting development of a new version of the pytest-easy-server project in its Git repo.

This section covers all variants of new versions:

• Starting a new major version (Mnew.0.0) based on the master branch

• Starting a new minor version (M.Nnew.0) based on the master branch

• Starting a new update version (M.N.Unew) based on the stable branch of its minor version

The description assumes that the andy-maier/pytest-easy-server Github repo is cloned locally and its upstream repo is assumed to have the Git remote name origin.

Any commands in the following steps are executed in the main directory of your local clone of the andy-maier/pytest-easy-server Git repo.

1. Set shell variables for the version that is being started and the branch it is based on:

   • MNU - Full version M.N.U that is being started

   • MN - Major and minor version M.N of that full version

   • BRANCH - Name of the branch the version that is being started is based on

   When starting a new major version (e.g. 1.0.0) based on the master branch:

   MNU=1.0.0
   MN=1.0
   BRANCH=master
   

   When starting a new minor version (e.g. 0.9.0) based on the master branch:

   MNU=0.9.0
   MN=0.9
   BRANCH=master
   

   When starting a new minor version (e.g. 0.8.1) based on the stable branch of its minor version:

   MNU=0.8.1
   MN=0.8
   BRANCH=stable_${MN}
   

2. Create a topic branch for the version that is being started:

   git checkout ${BRANCH}
   git pull
   git checkout -b start_${MNU}
   

3. Edit the version file:

   vi pytest_easy_server/_version.py
   

   and update the version to a draft version of the version that is being started:

   __version__ = 'M.N.U.dev1'
   

4. Edit the change log:

   vi docs/changes.rst
   

   and insert the following section before the top-most section:

   Version M.N.U.dev1
   ------------------
   
   This version contains all fixes up to version M.N-1.x.
   
   Released: not yet
   
   **Incompatible changes:**
   
   **Deprecations:**
   
   **Bug fixes:**
   
   **Enhancements:**
   
   **Cleanup:**
   
   **Known issues:**
   
   * See `list of open issues`_.
   
   .. _`list of open issues`: https://github.com/andy-maier/pytest-easy-server/issues
   

5. Commit your changes and push them to the remote repo:

   git status  # Double check the changed files
   git commit -asm "Start ${MNU}"
   git push --set-upstream origin start_${MNU}
   

6. On GitHub, create a Pull Request for branch start_M.N.U.

   Important: When creating Pull Requests, GitHub by default targets the master branch. When starting a version based on a stable branch, you need to change the target branch of the Pull Request to stable_M.N.

7. On GitHub, create a milestone for the new version M.N.U.

   You can create a milestone in GitHub via Issues -> Milestones -> New Milestone.

8. On GitHub, go through all open issues and pull requests that still have milestones for previous releases set, and either set them to the new milestone, or to have no milestone.

9. On GitHub, once the checks for the Pull Request for branch start_M.N.U have succeeded, merge the Pull Request (no review is needed). This automatically deletes the branch on GitHub.

10. Update and clean up the local repo:

    git checkout ${BRANCH}
    git pull
    git branch -d start_${MNU}
    

Appendix

Glossary

string

a unicode string or a byte string

unicode string

a Unicode string type (unicode in Python 2, and str in Python 3)

byte string

a byte string type (str in Python 2, and bytes in Python 3). Unless otherwise indicated, byte strings in this project are always UTF-8 encoded.

References

Python glossary

• Python 2.7 Glossary

• Python 3.9 Glossary

Change log

Version 0.6.0

Released: 2021-04-03

Initial release.