====== Reftests ======
A //reftest// is a test that compares the visual output of one file (the testcase) with the output of one or more other files (the references). Unlike the standard [[test:selftest|self-describing tests]], reftests can be scripted to run and report results automatically.
A test can be both a self-describing test and a reftest at the same time.
Here is an example of a reftest that is also a self-describing test: **[[http://test.csswg.org/source/contributors/apple/incoming/css-2d-transforms/rotate-90deg-001.xht|TEST]] & [[http://test.csswg.org/source/contributors/apple/incoming/css-2d-transforms/reftest/rotate-90deg-001-ref.xht|REF]]**
===== Components of a Reftest =====
A reftest has three parts:
; test file : The test file. This must follow the [[test:css2.1:format|CSS test format guidelines]].
; reference file : This is a different, usually simpler, file that results in the same rendering as the test. The reference file must not use the same features that are being tested. Sometimes more than one reference file is required.
; reftest comparison : A reference link or a line in a reftest "manifest" file that says which files are to be compared.
==== The Reftest Test ====
The test file uses the technology to be tested. This file must follow the [[test:css2.1:format|CSS test format guidelines]].
In addition to matching a reftest reference, the test may also function as an self-describing test. This is preferred because having the description lets the tester check that the test and the reference are not //both// being rendered incorrectly and triggering a false pass, and because designing it for an obvious fail makes it easier to find what went wrong when the reftest does fail.
==== The Reftest Reference ====
The reference file uses a different method to produce the same rendering as the test file. Multiple tests can (and often do) share the same reference file.
References should be named after the earliest test that uses them in the ''test-topic'' series they belong to, and must have either ''-ref'' or ''-notref'' appended to the file name just before the file extension. Depending on the test suite, they may be placed in the ''reference'' subdirectory of the main test directory or directly in the main test directory. If they are placed in the ''reference'' subdirectory then the ''-ref'' or ''-notref'' suffix may be omitted from their filename.
In some cases when creating the reference file, it is necessary to use features that, although different from the tested features, may themselves fail in such a manner as to cause the reference to render identically to a failed test. When this is the case, in order to reduce the possibility of false positive testing outcomes, multiple reference files should be used, each using a different technique to render the reference. One possibility is to create one or more references that must **not** match the test file, i.e.: a file that renders in the same manner as a failed test.
In other cases, the specification under test may allow multiple possible renderings. In this situation references must be supplied for each allowed rendering.
For example, if two self-describing tests ''list-style-type-003.xht'' and ''list-style-type-004.xht'' share the same reference, that reference could be named ''reftest/list-style-type-003-ref.xht''.
=== Simplified Reference Format ===
Like the format for the test file, the reftest reference format is also XHTML in UTF-8 with bitmap images in PNG format. Unlike the format for the test file, there is no metadata except for the [[test:css2.1:format#credits|author credits]] and optional [[test:css2.1:format#reference-links|reference links]].
CSS Reftest Reference
CONTENT OF REFERENCE
=== Common References ===
There are several common references, such as those used for parsing and selectors tests. Their names begin with ''ref-'' so they can be easily found in the ''reftest'' directory. Email public-css-testsuite@w3.org if you would like to add to the common references collection.
==== The Reftest Comparison ====
=== Reference Links ===
In order to designate which files are to be compared to the test file, and the nature of the comparison, the test file must have one or more links to the reference files as described in the [[test:css2.1:format#reference-links|test format]]. If multiple reference files must be matched, each reference file should, in turn, link to the next reference.
=== The Reftest Manifest ===
The use of reftest manifest files in the test source is deprecated in favor of reference links. The test build process produces reftest manifests as needed for input into testing tools.
The reftest manifest is a plain text file that lists test-reference pairs for comparison. Each line starts with ''=='' to indicate equality or ''!='' to indicate inequality. (Mozilla has a more sophisticated manifest format: we are not using its advanced features here at W3C.) This is followed by a space, the relative path to the test file, another space, and the relative path to the reference.
If a test has multiple ''=='' references then any of the references may match the test. If a test has multiple ''!='' references, then none of those references may match the test. The reference file may also have entries in the manifest, in this case, the renderings of the references must match each other as well in order to consider the test as passed.
White space followed by ''#'' indicates the start of a comment that runs until the end of the line. A line starting with ''#'' is also a comment.
The reftest manifest should be named ''reftest.list'' and placed in the ''reference'' subdirectory of the main test directory.
Here is an example of a reftest manifest.
# reftest.list snippet
== ../test-topic-000.xht test-topic-000-ref.xht
== ../test-topic-001.xht test-topic-001-ref.xht
== ../test-topic-002.xht test-topic-000-ref.xht # note same reference as test 000