TestCase

TestCase is the basic test case class for the HILSTER Testing Framework that should be used as a super class for a specialized test framework.

digraph inheritanceb4618a295e { rankdir=LR; size="8.0, 12.0"; "htf.core.TestCase" [fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5)",tooltip="``TestCase`` is the basic test case class for the HILSTER Testing Framework"]; "unittest.case.TestCase" -> "htf.core.TestCase" [arrowsize=0.5,style="setlinewidth(0.5)"]; "htf.testcase.TestCase" [fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5)",tooltip="``TestCase`` is the basic test case class for the HILSTER Testing Framework"]; "htf.core.TestCase" -> "htf.testcase.TestCase" [arrowsize=0.5,style="setlinewidth(0.5)"]; "unittest.case.TestCase" [fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5)",tooltip="A class whose instances are single test cases."]; }
class htf.TestCase(methodName='runTest')

TestCase is the basic test case class for the HILSTER Testing Framework that should be used using inheritance.

Parameters

methodName='runTest' (str) – the name of the test method to be run by the test runner.

addCleanup(function, *args, **kwargs)

Add a function, with arguments, to be called when the test is completed. Functions added are called on a LIFO basis and are called after tearDown on test failure or success.

Cleanup items are called even if setUp fails (unlike tearDown).

addTypeEqualityFunc(typeobj, function)

Add a type specific assertEqual style function to compare a type.

This method is for use by TestCase subclasses that need to register their own type equality functions to provide nicer error messages.

Parameters
  • typeobj – The data type to call this function on when both values are of the same type in assertEqual().

  • function – The callable taking two arguments and an optional msg= argument that raises self.failureException with a useful error message when the two arguments are not equal.

afterTest()

This method is called after a test ends and before htf.unittest.testcase.TestCase.tearDown() is called. This method is injected using the htf.decorators.after() decorator to be run in the test context.

If you override this method, make sure to call the original implementation:

class SpecializedTestCase(TestCase):

    def afterTest(self):
        # your code goes here
        super(TestCase, self).afterTest()
assertAlmostEqual(first, second, places=None, msg=None, delta=None)

Fail if the two objects are unequal as determined by their difference rounded to the given number of decimal places (default 7) and comparing to zero, or by comparing that the difference between the two objects is more than the given delta.

Note that decimal places (from zero) are usually not the same as significant digits (measured from the most significant digit).

If the two objects compare equal then they will automatically compare almost equal.

assertCountEqual(first, second, msg=None)

An unordered sequence comparison asserting that the same elements, regardless of order. If the same element occurs more than once, it verifies that the elements occur the same number of times.

self.assertEqual(Counter(list(first)),

Counter(list(second)))

Example:
  • [0, 1, 1] and [1, 0, 1] compare equal.

  • [0, 0, 1] and [0, 1] compare unequal.

assertEqual(first, second, msg=None)

Fail if the two objects are unequal as determined by the ‘==’ operator.

assertFalse(expr, msg=None)

Check that the expression is false.

assertGreater(a, b, msg=None)

Just like self.assertTrue(a > b), but with a nicer default message.

assertGreaterEqual(a, b, msg=None)

Just like self.assertTrue(a >= b), but with a nicer default message.

assertIn(member, container, msg=None)

Just like self.assertTrue(a in b), but with a nicer default message.

assertIs(expr1, expr2, msg=None)

Just like self.assertTrue(a is b), but with a nicer default message.

assertIsInstance(obj, cls, msg=None)

Same as self.assertTrue(isinstance(obj, cls)), with a nicer default message.

assertIsNone(obj, msg=None)

Same as self.assertTrue(obj is None), with a nicer default message.

assertIsNot(expr1, expr2, msg=None)

Just like self.assertTrue(a is not b), but with a nicer default message.

assertIsNotNone(obj, msg=None)

Included for symmetry with assertIsNone.

assertLess(a, b, msg=None)

Just like self.assertTrue(a < b), but with a nicer default message.

assertLessEqual(a, b, msg=None)

Just like self.assertTrue(a <= b), but with a nicer default message.

assertListEqual(list1, list2, msg=None)

A list-specific equality assertion.

Parameters
  • list1 – The first list to compare.

  • list2 – The second list to compare.

  • msg – Optional message to use on failure instead of a list of differences.

assertLogs(logger=None, level=None)

Fail unless a log message of level level or higher is emitted on logger_name or its children. If omitted, level defaults to INFO and logger defaults to the root logger.

This method must be used as a context manager, and will yield a recording object with two attributes: output and records. At the end of the context manager, the output attribute will be a list of the matching formatted log messages and the records attribute will be a list of the corresponding LogRecord objects.

Example:

with self.assertLogs('foo', level='INFO') as cm:
    logging.getLogger('foo').info('first message')
    logging.getLogger('foo.bar').error('second message')
self.assertEqual(cm.output, ['INFO:foo:first message',
                             'ERROR:foo.bar:second message'])
assertMultiLineEqual(first, second, msg=None)

Assert that two multi-line strings are equal.

assertNotAlmostEqual(first, second, places=None, msg=None, delta=None)

Fail if the two objects are equal as determined by their difference rounded to the given number of decimal places (default 7) and comparing to zero, or by comparing that the difference between the two objects is less than the given delta.

Note that decimal places (from zero) are usually not the same as significant digits (measured from the most significant digit).

Objects that are equal automatically fail.

assertNotEqual(first, second, msg=None)

Fail if the two objects are equal as determined by the ‘!=’ operator.

assertNotIn(member, container, msg=None)

Just like self.assertTrue(a not in b), but with a nicer default message.

assertNotIsInstance(obj, cls, msg=None)

Included for symmetry with assertIsInstance.

assertNotRegex(text, unexpected_regex, msg=None)

Fail the test if the text matches the regular expression.

assertRaises(expected_exception, *args, **kwargs)

Fail unless an exception of class expected_exception is raised by the callable when invoked with specified positional and keyword arguments. If a different type of exception is raised, it will not be caught, and the test case will be deemed to have suffered an error, exactly as for an unexpected exception.

If called with the callable and arguments omitted, will return a context object used like this:

with self.assertRaises(SomeException):
    do_something()

An optional keyword argument ‘msg’ can be provided when assertRaises is used as a context object.

The context manager keeps a reference to the exception as the ‘exception’ attribute. This allows you to inspect the exception after the assertion:

with self.assertRaises(SomeException) as cm:
    do_something()
the_exception = cm.exception
self.assertEqual(the_exception.error_code, 3)
assertRaisesRegex(expected_exception, expected_regex, *args, **kwargs)

Asserts that the message in a raised exception matches a regex.

Parameters
  • expected_exception – Exception class expected to be raised.

  • expected_regex – Regex (re pattern object or string) expected to be found in error message.

  • args – Function to be called and extra positional args.

  • kwargs – Extra kwargs.

  • msg – Optional message used in case of failure. Can only be used when assertRaisesRegex is used as a context manager.

assertRegex(text, expected_regex, msg=None)

Fail the test unless the text matches the regular expression.

assertSequenceEqual(seq1, seq2, msg=None, seq_type=None)

An equality assertion for ordered sequences (like lists and tuples).

For the purposes of this function, a valid ordered sequence type is one which can be indexed, has a length, and has an equality operator.

Parameters
  • seq1 – The first sequence to compare.

  • seq2 – The second sequence to compare.

  • seq_type – The expected datatype of the sequences, or None if no datatype should be enforced.

  • msg – Optional message to use on failure instead of a list of differences.

assertSetEqual(set1, set2, msg=None)

A set-specific equality assertion.

Parameters
  • set1 – The first set to compare.

  • set2 – The second set to compare.

  • msg – Optional message to use on failure instead of a list of differences.

assertSetEqual uses ducktyping to support different types of sets, and is optimized for sets specifically (parameters must support a difference method).

assertTrue(expr, msg=None)

Check that the expression is true.

assertTupleEqual(tuple1, tuple2, msg=None)

A tuple-specific equality assertion.

Parameters
  • tuple1 – The first tuple to compare.

  • tuple2 – The second tuple to compare.

  • msg – Optional message to use on failure instead of a list of differences.

assertWarns(expected_warning, *args, **kwargs)

Fail unless a warning of class warnClass is triggered by the callable when invoked with specified positional and keyword arguments. If a different type of warning is triggered, it will not be handled: depending on the other warning filtering rules in effect, it might be silenced, printed out, or raised as an exception.

If called with the callable and arguments omitted, will return a context object used like this:

with self.assertWarns(SomeWarning):
    do_something()

An optional keyword argument ‘msg’ can be provided when assertWarns is used as a context object.

The context manager keeps a reference to the first matching warning as the ‘warning’ attribute; similarly, the ‘filename’ and ‘lineno’ attributes give you information about the line of Python code from which the warning was triggered. This allows you to inspect the warning after the assertion:

with self.assertWarns(SomeWarning) as cm:
    do_something()
the_warning = cm.warning
self.assertEqual(the_warning.some_attribute, 147)
assertWarnsRegex(expected_warning, expected_regex, *args, **kwargs)

Asserts that the message in a triggered warning matches a regexp. Basic functioning is similar to assertWarns() with the addition that only warnings whose messages also match the regular expression are considered successful matches.

Parameters
  • expected_warning – Warning class expected to be triggered.

  • expected_regex – Regex (re pattern object or string) expected to be found in error message.

  • args – Function to be called and extra positional args.

  • kwargs – Extra kwargs.

  • msg – Optional message used in case of failure. Can only be used when assertWarnsRegex is used as a context manager.

attachFile(filename, title)

Attach a file to the current test that is put into the test report.

The attachment size is checked using htf.TestCase.get_maximum_attachment_size().

Parameters
  • filename (str) – the filename to be attached

  • title (str) – the title

attachUrl(url, title)

Attach an url to the current test that is put into the test report. This results in an external link and should be used for huge files.

Parameters
  • url (str) – the url to be attached

  • title (str) – the title

attach_file(filename, title)

Attach a file to the current test that is put into the test report.

The attachment size is checked using htf.TestCase.get_maximum_attachment_size().

Parameters
  • filename (str) – the filename to be attached

  • title (str) – the title

attach_url(url, title)

Attach an url to the current test that is put into the test report. This results in an external link and should be used for huge files.

Parameters
  • url (str) – the url to be attached

  • title (str) – the title

debug()

Run the test without collecting errors in a TestResult

doCleanups()

Execute all cleanup functions. Normally called for you after tearDown.

expect(expectedValue, observedValue, errorMessage=None)

Expect that expectedValue and observedValue are the same. If expectedValue and observedValue differ an AssertionError is raised.

Parameters
  • expectedValue – the expected value.

  • observedValue – the observed value that is compared to expectedValue.

  • errorMessage=None (str) – an optional message.

expectStep(title, expectedValue, observedValue, errorMessage=None)

Creates a test step in test report and on stdout using title. Expect that expectedValue and observedValue are the same. If expectedValue and observedValue differ an AssertionError is raised.

Parameters
  • title (str) – the step’s title.

  • expectedValue – the value that is expected.

  • observedValue – the current observed value that is compared to expectedValue.

  • errorMessage=None (str) – an optional message.

fail(msg=None)

Fail immediately, with the given message.

getDoc()

Returns the concatination of test case docs and test method docs.

get_maximum_attachment_size()

Return the maximum attachments size in bytes.

The default size is 10 Mb.

Returns

the maximum attachment size in bytes

Return type

int

log(*args)

Log *args to stdout and into the result of the given step if run in a step context block. This method is not thread-safe!

Parameters

*args (iterable) – a tuple of python objects that must be convertible into a string. All parameters are concatenated with spaces and converted to string before.

runBackground(func, name=None, joinTimeout=1, args=None, kwargs=None)

With this method you can run another method in background.

If this method uses a while loop, it must poll self._runThreadsEnable to be able to be terminated. If the background thread does not terminate, the foreground thread never ends!

Parameters
  • func (callable) – the method to be run in background.

  • name=None (str) – a name for the background task. Default: None. If name is None func.__name__ is used.

  • joinTimeout=1 (float) – the timeout to be used when calling join on the thread.

  • args=None (tuple) – a tuple of positional arguments that are passed to the background task when it is called.

  • kwargs=None (dict) – a dictionary of keyword arguments that are passed to the background task when it is called.

runPeriodic(func, period, maximumPeriod=None, name=None, raiseException=True, args=None, kwargs=None)

With this method you can run another method periodically in background with a given period.

Parameters
  • func (callable) – the method to be called periodically in background.

  • period (float) – the period in second.

  • maximumPeriod=None (float) – if set to a float >= period func may take up to maximumPeriod time without raising an exception. A warning is printed out to stdout instead.

  • name=None (str) – a name for the background task.

  • raiseExceptions=True – if set to True exceptions are put into the exception queue to let a test fail if a background thread fails.

  • args=None (tuple) – a tuple of positional arguments that are passed to the background task when it is called.

  • kwargs=None (dict) – a dictionary of keyword arguments that are passed to the background task when it is called.

Returns

a callable object is returned that stops

execution when called.

Return type

stop-trigger (callable)

setMetadata(name, value)

Set metadata for a running test.

Parameters
  • name (str) – the name of the metadata

  • value (str) – the value of the metadata (value is casted to a string automatically)

Raises

AssertionError – if name was already set to non accidently overwrite existing metdata

setOutputCaptureEnable(enabled=True)

Enables or disables capturing of stdout and stderr for later use in test reports.

Parameters

enabled=True (bool) – the new enabled state

setUp()

This method is called from test runner before a test is executed to create the test-fixtures.

If you override this method, make sure to call the original implementation:

def setUp(self):
    super(TestCase, self).setUp()
     # your code goes here
classmethod setUpClass()

Hook method for setting up class fixture before running tests in the class.

set_metadata(name, value)

Set metadata for a running test.

Parameters
  • name (str) – the name of the metadata

  • value (str) – the value of the metadata (value is casted to a string automatically)

Raises

AssertionError – if name was already set to non accidently overwrite existing metdata

shortDescription()

Returns the concatination of test case docs and test method docs.

skipStep(message)

Skip the current test step.

skipTest(message)

Skip the current test.

skip_step(message)

Skip the current test step.

skip_test(message)

Skip the current test.

step(title, callableObj=None, *args, **kwargs)

Run a test step that is documented in the test report.

Parameters
  • title (str) – the step’s title.

  • callableObj=None (callable) – a callable object that is called within the step or None.

  • *args (tuple) – a tuple of positional arguments for the step’s context.

  • **kwargs (dict) – a dictionary of keyword arguments for the step’s context.

Usage:

with self.step(title="Description of this test step .."):
    # methodCall()
    self.assertTrue(True "Message in case of an error")

Or an equivalent approach:

self.step(title="Description of this test step comes here ..",
    self.assertTrue, True, "Message in case of an error")
# method calls are not possible here!
stopThread(nameOrFunction)

This method stops a specific thread with the name or function described by nameOrFunction by calling join() with join timeout supplied by runPeriodic() or runBackground() or on it. It does not set a stop condition so the thread must end and must not include an endless loop. If the thread is still alive after the first join messages are printed to inform the user.

Parameters

nameOrFunction (str, callable) – the name or the function (reference) of the thread to be stopped.

Returns

the number of stopped threads.

Return type

int

Raises

Exception – if the thread was not found

stopThreads()

This method stops all background threads by setting self._runThreadsEnable to False and calling thread.join() until it is not alive anymore.

subTest(**kwds)

Return a context manager that will return the enclosed block of code in a subtest identified by the optional message and keyword parameters. A failure in the subtest marks the test case as failed but resumes execution at the end of the enclosed block, allowing further test code to be executed.

tearDown()

This method is called from test runner after a test was executed regardless of it’s result.

This method shall destroy all used fixtures.

If you override this method, make sure to call the original implementation:

def tearDown(self):
    # your code goes here
    super(TestCase, self).tearDown()
classmethod tearDownClass()

Hook method for deconstructing the class fixture after running all tests in the class.

Example for a specialized test case:

Example:

class SpecializedTestCase(TestCase):

    def setUp(self):
        TestCase.setUp(self)

        # your code comes here

    def tearDown(self):
        # your code comes here

        TestCase.tearDown(self)

To create test cases in your specialized test framework you should again subclass the specialized test case like this:

class ASpecializedTest(SpecializedTestCase):
    """
    The test can be documented in reStructuredText markup which
    will be shown in the test reports.
    """

    def test_whatEver(self):
        """
        This test can have a comment in reStructuredText markup, too which
        will be shown in the test reports as well.
        """
        self.assertTrue(True)

    def test_anotherTestStep(self):
        """no comment"""
        self.assertTrue(True)

Examples to document test steps:

Test steps are automatically documented in the test report using the step() context block:

class Test(TestCase):
    def test_simpleStep(self):
        for ii in range(10):
            with self.step(title="step %i" % ii):
                self.assertTrue(True)

    def test_anotherSimpleStep(self):
        result = False
        self.step("Check that the result is True", self.assertTrue, result, "The result is not True")

    def test_longStep(self):
        with self.step(title="multiple assertions in one step"):
            self.assertTrue(True)
            self.assertFalse(False)
            # something else..
            self.assertTrue(result)

Examples for thread usage:

A background thread:

class TestTestCase(TestCase):
    def test_threadEndsNormally(self):
        """
        A thread is run in background that terminates normally.
        """
        def thread():
            print("thread starts")
            time.sleep(1)
            print("thread ends")
        self.runBackground(func=thread, name="thread", joinTimeout=2)

A background thread that ends later:

class TestTestCase(TestCase):
    def test_threadEndsLater(self):
        """
        A thread is run in background that terminates later.
        """
        def thread():
            print("thread starts")
            time.sleep(1)
            print("thread ends")
        self.runBackground(func=thread, name="thread", joinTimeout=.1)

A background thread with a while loop:

class TestTestCase(TestCase):
    def test_threadWithWhileLoop(self):
        """
        A thread is run in background with a while loop
        polling the `self._runThreadsEnable` variable.
        """
        def thread(*args, **kwargs):
            print("thread:", args, kwargs)
            print("thread starts")
            while self._runThreadsEnable:
                print("thread active")
                time.sleep(1)
                print("thread inactive")
            print("thread ends")
        self.runBackground(func=thread, name="thread", joinTimeout=1, args=(1, 2, 3,), kwargs={'blah' : 'blub'})
        time.sleep(3.0)

Periodic background execution:

class TestTestCase(TestCase):
    def test_runPeriodic(self):
        """
        A method is run in background periodically.
        """
        self._counter = 0

        def function(*args, **kwargs):
            print("function()")
            self._counter += 1

        stop = self.runPeriodic(func=function, period=1.0, name="periodic function", raiseException=True, args=(), kwargs={})
        time.sleep(2.5)
        self.assertTrue(self._counter == 3)

        stop()
        time.sleep(2.5)
        self.assertTrue(self._counter == 3)

Periodic background execution with parameters:

class TestTestCase(TestCase):
    def test_periodicAndBackgroundParameters(self):
        """
        A method is run in background peridically with parameters.
        """
        self._args = None
        self._kwargs = None

        def function(*args, **kwargs):
            print("function()")
            self._args = args
            self._kwargs = kwargs

        args = (1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
        kwargs= {"foo" : "bar"}

        self.runPeriodic(func=function, period=1.0, name="periodic function", raiseException=True, args=args, kwargs=kwargs)
        time.sleep(2.5)

        print(self._args)
        print(self._kwargs)
        self.assertTrue(self._args == args)
        self.assertTrue(self._kwargs == kwargs)

A background thread that raises an exception:

class TestTestCase(TestCase):
    @expectedFailure
    def test_exceptionInBackgroundTask(self):
        """
        An exeption is raised in a background tasks that leads to an exception
        in the foreground task.
        """
        def function():
            raise Exception("Das ist ein Test!")

        self.runBackground(func=function, name="function")
        time.sleep(1.0)
        # tearDown() must be called here because it always leads to an Exception if run from TestRunner context!
        self.tearDown()