Browseable Test Harness

In 2007, Dominique Hazael-Massieux created a CGI-based harness for the Mobile Web Initiative's test suites. One of the very neat things about the MWI test harness is that it associates the results with a user agent string. This means there's no need for users to log in or to select their UA. They just load a test and click Pass/Fail/Can't Tell.

An easy-to-use harness for reporting tests like this would be useful across all of W3C's Interaction Domain working groups. It has the potential to make the process of creating implementation reports much, much easier, and it would allow more involvement from the community outside W3C.

The CSS Working Group planned to extend this harness and use it for the CSS conformance tests and implementation reports. HP took Dominique's source code, created a more flexible back-end and implementing some of the improvements we felt were needed. Dominique posted the results of HP's work on the W3C site, along with a working prototype. But HP then disengaged from the project after a company reorganization.

The current state of the project is that nobody here knows how the harness works, whether it's reliable, or how to set it up with a new set of tests. The reporting half of the improvements were not implemented, and the output is mostly unhelpful in its current state.

The source code, bug database, and prototype URL are all listed below. If you are interested in taking ownership of this project, and have any questions, please contact the public-css-testuite or public-test-infra mailing list.

• Current Harness for CSS2.1
• MWI's harness
• MWI's harness source code
• Harness prototype
• Harness source code
• Harness bug database

These are the improvements we'd like to see, numbered roughly by priority.

• (1) Harness using <object> to contain the test and pass/fail buttons on the containing page rather than inside the test file. (This format is good for browsers.) The <object> will contain a hypertext link to the test as fallback.

• (2) Harness using links to open the test and pass/fail buttons on the page containing the link rather than inside the test file. This format will also allow the tester to enter the UA string manually, for cases where the UA being tested is non-interactive. (This format is necessary for printers.)

In both cases the test itself should be referenced as a link, not fed through the CGI script. (This avoids tampering with the HTTP headers that normally get served up with the tests.)

• (8) It would be nice if these harnesses could include some meta information about the test in addition to the buttons. E.g. the test ID (filename before extension), test title, any requirements documented in the test (“Warning: Must install Ahem font.” etc). (We can extract this information into a flat-file database during the test suite build process so the harness doesn't have to do any analysis of the tests themselves.)

• (3) Ability to consolidate results for various user agent strings under one category name. E.g. consolidate results for all UA strings that represent Opera 9.25 Beta 1 regardless of OS and localization.

• (4) Pass/fail scores for the whole test suite

• (5) Ability to report consolidated pass/fail scores for a named groups of tests. (To create e.g. a summary of what features are supported and to what level.)

• (6) Interface for generating reports based on various parameters. URLs to these reports should be short and clean so they can be passed around in blogs/IM/email etc.

• (9) Prettier reports. :)

• (7) Ability to delete all data for a given test (so that when a test is changed we can invalidate the results for that test). This process could be triggered manually on the command line rather than via CGI—that avoids the need for a login system.

• Put requirements up front
• Add Skip button
• Track session so that a bad user's data can be deleted