Data Driven Testing

Data Driven Testing allows dynamic generation of tests using a data source. The data source can be Python objects, JSON, CSV or YAML files.

To use Data Driven Testing simply decorate tests using the data or combinatoric decorators as mentioned below.

htf.TestCase will automatically generate tests when it is instantiated. The generated method names and docstrings contain the specific parameters used by the test.

Code-based Examples

Full Example

The following example dynamically generates three test methods from a list of scalars. To use data driven testing in htf the test methods are decorated. The decorator supplies the data. The general structure using a Python list as data input looks as follows:

import htf

class Tests(htf.TestCase):
    @htf.data([1.0, 2.0, 3.0])
    def test_scalar(self, value):
        print("value:", value)

if __name__ == "__main__":
    htf.main()

List of Scalars

The following examples demonstrate the usage of Python objects as data input. The decorator to be used when the data is supplied as a Python object is @htf.data(<object>).

The following example uses a list of integers as data.

@htf.data([23, 24, 25])
def test_scalar(self, value):
    print("value:", value)

# generates
# test_scalar_23(self, value=23)
# test_scalar_24(self, value=24)
# ...

Iterator

An iterator can also be supplied.

@htf.data(iter([23, 24, 25]))
def test_iterator(self, value):
    print("value:", value)

Generator

It is also possible to supply a generator.

def generator():
    yield 10
    yield 20
    yield 30

@htf.data(generator())
def test_generator_instance(self, value):
    print("value:", value)

# generates
# test_generator_instance_10(self, value=10)
# test_generator_instance_20(self, value=20)
# ...

@htf.data(generator)
def test_generator(self, value):
    print("value:", value)

# generates
# test_generator_10(self, value=10)
# test_generator_20(self, value=20)
# ...

List of Dicts

The following example supplies a list of dictionaries. The number of parameters must match the number of dictionary entries.

@htf.data([dict(a=23, b=1), dict(a=24, b=2), dict(a=25, b=3)])
def test_dict(self, a, b):
   print("a:", a, "b:", b)

# generates
# test_dict_a_23_b_1(self, a=23, b=1)
# test_dict_a_24_b_2(self, a=24, b=2)
# ...

List of Lists

Lastly it is also possible to supply lists of lists. When called, value contains a list.

@htf.data([[1, 2, 3], [4, 5, 6]])
def test_list_of_int(self, value):
    print("value:", value)

# generates
# test_list_of_int_1_2_3(self, value=[1, 2, 3])
# ...

@htf.data([[dict(value=23)], [dict(value=24)], [dict(value=25)]])
def test_list_of_lists_of_dict(self, values):
   print("value:", values)

# generates
# test_list_of_lists_of_dict_value_23(self, value=23)
# test_list_of_lists_of_dict_value_23(self, value=24)
# ...

Unpack the content of value by setting unpack=True to distribute the list entries to three parameters.

@htf.data([[1, 2, 3], [4, 5, 6]], unpack=True)
def test_list_of_int_unpack(self, a, b, c):
    print("a:", a, "b:", b, "c:", c)

# generates
# test_list_of_int_unpack_1_2_3(self, a=1, b=2, c=3)
# ...

File-based Examples

The following examples demonstrate the usage of various file formats as data input. The supported file formats and their corresponding decorators are:

File type

Decorator

JSON-file

@htf.json_data("filename.json")

YAML-file

@htf.yaml_data("filename.yaml")

CSV-file

@htf.csv_data("filename.csv")

In each case there is also the option of first creating an iterator for the file content which is handed to the @htf.data() decorator. For this purpose the following syntax is used:

File type

Decorator and Iterator

JSON-file

@htf.data(htf.JSONFileIterator("filename.json"))

YAML-file

@htf.data(htf.YAMLFileIterator("filename.yaml"))

CSV-file

@htf.data(htf.CSVFileIterator("filename.csv"))

Furthermore, decorators for JSON and YAML input have the option unpack set to False by default. If changed to True lists will be unpacked as demonstrated in the below examples. There is no such option for the CSV decorator.

JSON Test Data

To supply simple scalar values as data from a JSON file, use the following code:

@htf.json_data("ddt_scalar.json")
def test_json_scalar(self, value):
    print("value: ", value)

# generates
# test_json_1337(self, value=1337)
# test_json_1338(self, value=1338)
# ...

@htf.data(htf.JSONFileIterator("ddt_scalar.json"))
def test_json_iterator(self, value):
    print("value:", value)

# generates
# test_json_iterator_1337(self, value=1337)
# test_json_iterator_1338(self, value=1338)
# ...

Contents of ddt_scalar.json:

[1337, 1338, 1339]

If the supplied data comes in form of nested lists, unpack can be set to True to assign each inner list element to a variable.

@htf.json_data("ddt_list.json", unpack=True)
def test_json_list(self, a, b, c):
    print("value: ", a, b, c)

# generates
# test_json_list_111_222_333(self, a=111, b=222, c=333)
# test_json_list_11_22_33(self, a=11, b=22, c=33)
# ...

@htf.json_data("ddt_list.json", unpack=False)
def test_json_list(self, value):
    print("value: ", value)

# generates
# test_json_list_111_222_333(self, value=[111, 222, 333])
# test_json_list_11_22_33(self, value=[11, 22, 33])
# ...

Contents of ddt_list.json:

[[1,2,3], [11,22,33], [111,222,333]]

YAML Test Data

Supplying scalar values as data from a YAML file works analogously to JSON files.

@htf.yaml_data("ddt_scalar.yaml")
def test_yaml_scalar(self, value):
    print("value:", value)

# generates
# test_yaml_1337(self, value=1337)
# test_yaml_1338(self, value=1338)
# ...

@htf.data(htf.YAMLFileIterator("ddt_scalar.yaml"))
def test_yaml_iterator(self, value):
    print("value:", value)

# generates
# test_yaml_iterator_1337(self, value=1337)
# test_yaml_iterator_1338(self, value=1338)
# ...

Contents of ddt_scalar.yaml:

[1337, 1338, 1339]

The following code showcases the use of the unpack option in the case of a YAML file.

@htf.yaml_data("ddt_list.yaml", unpack=True)
def test_yaml_list(self, a, b, c):
    print("value: ", a, b, c)

# generates
# test_yaml_list_111_222_333(self, a=111, b=222, c=333)
# test_yaml_list_11_22_33(self, a=11, b=22, c=33)
# ...

@htf.yaml_data("ddt_list.yaml", unpack=False)
def test_yaml_list(self, value):
    print("value: ", value)

# generates
# test_yaml_list_111_222_333(self, value=[111, 222, 333])
# test_yaml_list_11_22_33(self, value=[11, 22, 33])
# ...

Contents of ddt_list.yaml:

[[1,2,3], [11,22,33], [111,222,333]]

CSV Test Data

Tests can also be generated using CSV data sources. The first line of the CSV data is a header. The parameter names of the decorated method must match the entries of the header, i.e. the fist line of each column must contain the name of the parameter to which the data is to be assigned. Each line is supplied as a dict. Note that the entries of the dict are strings by default.

@htf.csv_data("ddt.csv")
def test_csv(self, a, b):
    print("a: ", a, "b: ", b)

# generates
# test_csv_a_1337_b_2222 (self, a=1337, b=2222)
# a: 1337 b: 2222
# test_csv_a_1338_b_3333 (self, a=1338, b=3333)
# a: 1338 b: 3333
# ...

@htf.data(htf.CSVFileIterator("ddt.csv"))
def test_csv(self, a, b):
    print("a: ", a, "b: ", b)

# generates
# test_csv_a_1337_b_2222 (self, a=1337, b=2222)
# a: 1337 b: 2222
# test_csv_a_1338_b_3333 (self, a=1337, b=3333)
# a: 1338 b: 3333
# ...

Contents of ddt.csv:

a,b
1337,2222
1338,3333
1339,4444

Combinatoric Data Generation

Combinatoric decorators can be used to manipulate the supplied data.

Cartesian Product

The product function produces every possible combination of items, taking one item of each given iterable.

@htf.product([1, 2, 3], repeat=2)
def test_product(self, a, b):
    print("a:", a, "b:", b)

# generates
# test_product_1_1(self, a=1, b=1)
# test_product_1_2(self, a=1, b=2)
# ...

@htf.product([1, 2], [3, 4], repeat=1)
def test_products(self, a, b):
    print("a:", a, "b:", b)

# generates
# test_products_1_3(self, a=1, b=3)
# test_products_2_3(self, a=2, b=3)
# ...

In combination with the aforementioned file iterator functions, this can be used to produce combinations of elements of given data files.

@htf.product(htf.JSONFileIterator("ddt_scalar.json"), repeat=2)
def test_json_products(self, a, b):
    print("a:", a, "b:", b)

# generates
# test_products_1337_1337(self, a=1337, b=1337)
# test_products_1337_1338(self, a=1337, b=1338)
# ...

@htf.product(list(range(10)), repeat=3)
def test_long_products(self, *args):
    print("args:", args)

# generates
# test_long_products_0_0_1(self, *(0, 0, 1))
# test_long_products_0_0_2(self, *(0, 0, 2))
# ...

Permutations

The permutations function creates all permutations of length r of a given iterable.

@htf.permutations([1, 2, 3], r=2)
def test_permutations(self, a, b):
    print("a:", a, "b:", b)

# generates
# test_permutations_1_2(self, a=1, b=2)
# test_permutations_1_3(self, a=1, b=3)
# ...

Combinations

The combinations function produces all possible combinations of elements of an iterable without changing their order. Elements are not combined with themselves.

@htf.combinations([1, 2, 3], r=2)
def test_combinations(self, a, b):
    print("a:", a, "b:", b)

# generates
# test_combinations_1_2(self, a=1, b=2)
# test_combinations_1_3(self, a=1, b=3)
# ...

The combinations_with_replacement function does the same as the combinations function, except elements are also combined with themselves.

@htf.combinations_with_replacement([1, 2, 3], r=2)
def test_combinations_with_replacement(self, a, b):
    print("a:", a, "b:", b)

# generates
# test_combinations_with_replacement_1_1(self, a=1, b=1)
# test_combinations_with_replacement_1_2(self, a=1, b=2)
# ...

Data Decorators

htf.data(iterable, unpack=False)

Create a data driven test using data.

Parameters
  • iterable – an iterable used as parameter data source. Can also be a callable.

  • unpack=False (bool) – if set to True iterables are unpacked.

htf.csv_data(filename)

Create a data driven test using data from a CSV file. The first line of the CSV data is a header. Each column from the first is used as the parameter name of the decorated method. Each line is supplied as a dict.

Parameters

filename (str) – the filename to be read.

htf.yaml_data(filename, unpack=False)

Create a data driven test using data from a YAML file.

Parameters
  • filename (str) – the filename to be read.

  • unpack=False (bool) – if set to True iterables are unpacked.

htf.json_data(filename, unpack=False)

Create a data driven test using data from a json file.

Parameters
  • filename (str) – the filename to be read.

  • unpack=False (bool) – if set to True iterables are unpacked.

Combinatoric Decorators

htf.product(*iterables, **repeat)

Create a data driven test using itertools.product() with a cartesian product of input *iterables.

Parameters
  • *iterables – iterables to be passed to itertools.product()

  • repeat=1 – number of repetitions.

htf.permutations(iterable, r=None)

Create a data driven test using itertools.permutations() with successive r length permutations of elements in the iterable.

Parameters
htf.combinations(iterable, r)

Create a data driven test using itertools.combinations() with successive r length subsequences of elements from iterable.

Parameters
htf.combinations_with_replacement(iterable, r)

Create a data driven test using itertools.combinations() with successive r length subsequences of elements from iterable allowing individual elements to be repeated more than once.

Parameters

File Iterators

class htf.CSVFileIterator(filename)

CSV file iterator.

Parameters

filename (str) – the filename to be read.

Returns

iterator for the file contents as dict.

Return type

iterator

class htf.YAMLFileIterator(filename)

YAML file iterator.

Parameters

filename (str) – the filename to be read.

Returns

iterator for the file contents.

Return type

iterator

class htf.JSONFileIterator(filename)

JSON file iterator.

Parameters

filename (str) – the filename to be read.

Returns

iterator for the file contents.

Return type

iterator