Script Tests

A script test is a test that uses JavaScript to programmatically verify one or more specific functionalities. Unlike self-describing tests and reftests, script tests do not verify rendering.

Script tests should be used to verify functionalities that do not require rendering, for example, a JavaScript API. Script tests can also be used to add automation to reftests for clients that do not support reftests.

Components of a Script Test

A script test can be written as a single file with a script block that contains the JavaScript, or the JavaScript can be contained in a separate .js file that is imported by the test file.

Test Harness

The test harness (written and maintained by James Graham) is a JavaScript file that facilitates writing test cases. Specifically, test harness offers test authors:

A convenient API for making common test assertions
Support for testing synchronous and asynchronous DOM features in a way that promotes clear, robust, tests

Script tests written for the CSSWG should use test harness whenever possible.

Basic Usage

The basic usage of the test harness (testharness.js) is described in the beginning of the file.

To use the harness, import both scripts into the test document per the script tag below. Note that an expectation is that all tests must be run-able in place in the W3C's test repository. As such, an absolute path to the script file must be used and the script file will be in resources directory at the root of the repository.

<script type="text/javascript" src="/resources/testharness.js"></script>
<script type="text/javascript" src="/resources/testharnessreport.js"></script>

Each test can have one or more asserts. Note: some prefer a test is limited to a single assert but that is not mandatory.
Each test is atomic in the sense that a single test has a single result (pass/fail/timeout).
Within each test one may have a number of asserts.
A test fails at the first failing assert, and the remainder of the test is (typically) not run.
If the file containing the tests is a HTML file with an element of id “log”, this will be populated with a table containing the test results after all the tests have run.

Writing tests

The full API of testharness.js is documented within the file itself. The API is also documented here.

Bug Reporting

The discussion forum for the test harness is the W3C's cross-WG public-test-infra mail list. This list is also used to report testharness.js bugs or bugs can also be directly added to the W3C's Bugzilla: Product = Testing; Component = testharness.js (James Graham is the “default assignee”).

Per-Test Metadata

Test-specific metadata can be passed in the properties object to the test constructor. These are used when an individual test has different metadata from that stored in the <head>. The recognized metadata properties are:

help - The url for the part of the specification being tested
assert - A human readable description of what the test is attempting to prove
author - Name and contact information for the author of the test in the format: “Name <email_addr>” or “Name http://contact/url”

Example:

test(function() { assert_true(true); },
     'test_name', 
     { help: 'http://www.w3.org/TR/spec#section', 
       assert: ['This tests something.', 'This also tests something else.'],
       author: ['John Doe <john@doe.com>', 'Jane Doe http://example.com/doe/jane'] }
);

Each value can be either a single string, or an array of strings if multiple values need to be specified. These values would override any metadata set in the <head> of the test and are only needed when the individual test's metadata is different from what's in the <head>.

If there is only a single script test in a file, all metadata should be in the <head> rather than the test constructor.

Metadata Cache

The metadata cache exposes the names of the script tests and all associated per-test metadata to our testing tool suite. Using this data we can track the testing coverage of each specification as well as generate testing statistics and reports. If there are multiple script tests in a file, the metadata cache must be present, even if there is no test-specific metadata present (to list the names of each script test).

The version of testharnessreport.js on http://test.csswg.org/resources now contains code that reads your per-test metadata and compares it to the cached version stored in the <head>. If the cache is not present, or is out of sync, it'll display a message to that effect and generate a link which, when clicked, will generate the appropriate source code for the cached metadata suitable for copy/paste into the test's <head>. The metadata cache need only be generated once just before submitting a test to suites that care about metadata, and regenerated only when the metadata changes.

To access this functionality, launch your script test in a browser and follow the link that appears at the top of the page.

Table of Contents