tests.test

Test framework for pyglet. Reads details of components and capabilities from a requirements document, runs the appropriate unit tests.

How to Run the Tests

python tests/test.py top app graphics clock resource # these all run automatically
python tests/test.py font media text
python tests/test.py image
python tests/test.py window

Because the tests are interactive, they can take quite a while to complete. The ‘window’ section in particular takes a long time. It can be frustrating to get almost through the tests and then something gets messed up, so we suggest you run the tests in sections as listed above. If you are curious, the sections are defined in tests/plan.txt.

Here are the different sections and how long they take.

Section Time to Run
top automatic
app automatic
graphics automatic
clock automatic
resource automatic
font 1 minute
media 1 minute
text 1 minute
image 5 minutes
window 10 minutes

Overview

First, some definitions:

Test case:
A single test, implemented by a Python module in the tests/ directory. Tests can be interactive (requiring the user to pass or fail them) or non-interactive (the test passes or fails itself).
Section:
A list of test cases to be run in a specified order. Sections can also contain other sections to an arbitrary level.
Capability:

A capability is a tag that can be applied to a test-case, which specifies a particular instance of the test. The tester can select which capabilities are present on their system; and only test cases matching those capabilities will be run.

There are platform capabilities “WIN”, “OSX” and “X11”, which are automatically selected by default.

The “DEVELOPER” capability is used to mark test cases which test a feature under active development.

The “GENERIC” capability signifies that the test case is equivalent under all platforms, and is selected by default.

Other capabilities can be specified and selected as needed. For example, we may wish to use an “NVIDIA” or “ATI” capability to specialise a test-case for a particular video card make.

Some tests generate regression images if enabled, so you will only need to run through the interactive procedure once. During subsequent runs the image shown on screen will be compared with the regression images and passed automatically if they match. There are command line options for enabling this feature.Literal block

By default regression images are saved in tests/regression/images/

Running tests

The test procedure is interactive (this is necessary to facilitate the many GUI-related tests, which cannot be completely automated). With no command-line arguments, all test cases in all sections will be run:

python tests/test.py

Before each test, a description of the test will be printed, including some information of what you should look for, and what interactivityLiteral block is provided (including how to stop the test). Press ENTER to begin the test.

When the test is complete, assuming there were no detectable errors (for example, failed assertions or an exception), you will be asked to enter a [P]ass or [F]ail. You should Fail the test if the behaviour was not as described, and enter a short reason.

Details of each test session are logged for future use.

Command-line options:

–plan=
Specify the test plan file (defaults to tests/plan.txt)
–test-root=
Specify the top-level directory to look for unit tests in (defaults to test/)
–capabilities=
Specify the capabilities to select, comma separated. By default this only includes your operating system capability (X11, WIN or OSX) and GENERIC.
–log-level=
Specify the minimum log level to write (defaults to 20: info)
–log-file=
Specify log file to write to (defaults to “pyglet.%d.log”)
–regression-capture
Save regression images to disk. Use this only if the tests have already been shown to pass.
–regression-check
Look for a regression image on disk instead of prompting the user for passage. If a regression image is found, it is compared with the test case using the tolerance specified below. Recommended only for developers.
–regression-tolerance=
Specify the tolerance when comparing a regression image. A value of 2, for example, means each sample component must be +/- 2 units of the regression image. Tolerance of 0 means images must be identical, tolerance of 256 means images will always match (if correct dimensions). Defaults to 2.
–regression-path=
Specify the directory to store and look for regression images. Defaults to tests/regression/images/
–developer
Selects the DEVELOPER capability.
–no-interactive=
Don’t write descriptions or prompt for confirmation; just run each test in succcession.

After the command line options, you can specify a list of sections or test cases to run.

Examples

python tests/test.py –capabilities=GENERIC,NVIDIA,WIN window

Runs all tests in the window section with the given capabilities. Test just the FULLSCREEN_TOGGLE test case without prompting for input (useful for development).

python tests/image/PIL_RGBA_SAVE.py

Run a single test outside of the test harness. Handy for development; it is equivalent to specifying –no-interactive.

Writing tests

Add the test case to the appropriate section in the test plan (plan.txt). Create one unit test script per test case. For example, the test for window.FULLSCREEN_TOGGLE is located at:

tests/window/FULLSCREEN_TOGGLE.py

The test file must contain:

  • A module docstring describing what the test does and what the user should look for.
  • One or more subclasses of unittest.TestCase.
  • No other module-level code, except perhaps an if __name__ == ‘__main__’ condition for running tests stand-alone.
  • Optionally, the attribute “__noninteractive = True” to specify that the test is not interactive; doesn’t require user intervention.

During development, test cases should be marked with DEVELOPER. Once finished add the WIN, OSX and X11 capabilities, or GENERIC if it’s platform independent.

Writing regression tests

Your test case should subclass tests.regression.ImageRegressionTestCase instead of unitttest.TestCase. At the point where the buffer (window image) should be checked/saved, call self.capture_regression_image(). If this method returns True, you can exit straight away (regression test passed), otherwise continue running interactively (regression image was captured, wait for user confirmation). You can call capture_regression_image() several times; only the final image will be used.

Python 3

The tests have to be processed by 2to3 in order to run them with Python 3.

This can be done with:

2to3 --output-dir=tests3 -W -n tests

And then run the tests int tests3 directory.

Table Of Contents

Previous topic

Pyglet documentation system