A reftest is a test that compares the visual output of one file (the testcase) with the output of one or more other files (the references). Unlike the standard self-describing tests, reftests can be scripted to run and report results automatically.
A test can be both a self-describing test and a reftest at the same time.
A reftest has three parts:
The test file uses the technology to be tested. This file must follow the CSS test format guidelines.
In addition to matching a reftest reference, the test may also function as an self-describing test. This is preferred because having the description lets the tester check that the test and the reference are not both being rendered incorrectly and triggering a false pass, and because designing it for an obvious fail makes it easier to find what went wrong when the reftest does fail.
The reference file uses a different method to produce the same rendering as the test file. Multiple tests can (and often do) share the same reference file.
References should be named after the earliest test that uses them in the test-topic series they belong to, and must have either -ref or -notref appended to the file name just before the file extension. Depending on the test suite, they may be placed in the reference subdirectory of the main test directory or directly in the main test directory. If they are placed in the reference subdirectory then the -ref or -notref suffix may be omitted from their filename.
In some cases when creating the reference file, it is necessary to use features that, although different from the tested features, may themselves fail in such a manner as to cause the reference to render identically to a failed test. When this is the case, in order to reduce the possibility of false positive testing outcomes, multiple reference files should be used, each using a different technique to render the reference. One possibility is to create one or more references that must not match the test file, i.e.: a file that renders in the same manner as a failed test.
In other cases, the specification under test may allow multiple possible renderings. In this situation references must be supplied for each allowed rendering.
list-style-type-003.xht and list-style-type-004.xht share the same reference, that reference could be named reftest/list-style-type-003-ref.xht.
Like the format for the test file, the reftest reference format is also XHTML in UTF-8 with bitmap images in PNG format. Unlike the format for the test file, there is no metadata except for the author credits and optional reference links.
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"> <head> <title>CSS Reftest Reference</title> <link rel="author" title="NAME_OF_AUTHOR" href="mailto:EMAIL OR http://CONTACT_PAGE"/> <style type="text/css"><![CDATA[ CSS FOR REFERENCE ]]></style> </head> <body> CONTENT OF REFERENCE </body> </html>
There are several common references, such as those used for parsing and selectors tests. Their names begin with ref- so they can be easily found in the reftest directory. Email public-css-testsuite@w3.org if you would like to add to the common references collection.
In order to designate which files are to be compared to the test file, and the nature of the comparison, the test file must have one or more links to the reference files as described in the test format. If multiple reference files must be matched, each reference file should, in turn, link to the next reference.
The use of reftest manifest files in the test source is deprecated in favor of reference links. The test build process produces reftest manifests as needed for input into testing tools.
The reftest manifest is a plain text file that lists test-reference pairs for comparison. Each line starts with == to indicate equality or != to indicate inequality. (Mozilla has a more sophisticated manifest format: we are not using its advanced features here at W3C.) This is followed by a space, the relative path to the test file, another space, and the relative path to the reference.
If a test has multiple == references then any of the references may match the test. If a test has multiple != references, then none of those references may match the test. The reference file may also have entries in the manifest, in this case, the renderings of the references must match each other as well in order to consider the test as passed.
White space followed by # indicates the start of a comment that runs until the end of the line. A line starting with # is also a comment.
The reftest manifest should be named reftest.list and placed in the reference subdirectory of the main test directory.
# reftest.list snippet == ../test-topic-000.xht test-topic-000-ref.xht == ../test-topic-001.xht test-topic-001-ref.xht == ../test-topic-002.xht test-topic-000-ref.xht # note same reference as test 000