Review Process Using Shepherd

This page describes the test submission and review process using Shepherd, Mercurial, and the mailing list. An alternative process using Github is documented on the W3C Test Resource Center.

Whether using this process or Github, test authors and reviewers must adhere to the W3C Test Review Policy.

The general process looks like this:

1. Pre-condition: Test authors and reviewers must be set up with Mercurial access. See the Quick Guide to Mercurial, including the section on obtaining write access.
2. When a test author is ready for tests to be reviewed, s/he must push them to the repository in a directory named submitted. The tests will then be present in Shepherd as Submitted for Review.
3. The test author should then ask for a review on the public-css-testsuite@w3.org mailing list.
4. When a reviewer comes forward, s/he should examine the tests for correctness with respect to the format and style guidelines. A short checklist is available to assist the reviewer and a more detailed CSS-specific checklist is also available.
5. The reviewer should then notify the mailing list of the test review. It is not necessary to include all of the review feedback in the mail, just a link to the test suite in Shepherd.
   1. If the test passes review: Go to the next step.
   2. If a test does not pass review:
      1. Its status should be set to Needs Work in Shepherd
      2. Optionally, the reviewer may use the Whiteboard field to tag the reason why the test doesn't pass review. See examples below on how to use this field.
      3. The reviewer must provide feedback to the submitter in the comment fields in Shepherd, including what is wrong with the test and what steps should be taken to correct any problems.
      4. Alternatively, if the changes requested are minor and don't significantly alter the original design of the test (i.e., metadata only changes), the reviewer may make the changes and get approval of those changes from the submitter.
      5. The tests should then be updated to incorporate the reviewers feedback and be re-submitted Mercurial. Ideally, the original test author does this but it may done be any other qualified person. Upon re-submission, the test status in Shepherd will change to Resubmitted for Review
      6. When the updated tests are submitted, another notification should be sent to the mailing list asking for review.
      7. A reviewer looks at the changes and decides if they satisfy the original review feedback. Likewise, ideally this is original reviewer, but may also be any other qualified person.
6. Once the test/changes has pass review, the reviewer should note their acceptance in the test and re-submit the files to Mercurial. The commit message should indicating the test passed review and will be attached to the test in Shepherd. The test status in Shepherd will then change to Accepted
7. The Owner : [This link should be exposed to all roles in Shepherd. Right now, only admins can see it] of that suite can either approve the reviewer's judgement if the reviewer is known to be competent in this area, or review the test himself. Once the Owner is satisfied the test can be set to Approved in Shepherd. (If the initial reviewer was an Owner then this step is automatic).

In some cases, tests that have been approved in the past are revisited and identified to have issues. When this occurs, this is also considered a review for which the process begins at Step 5 above.

The whiteboard field in Shepherd is exposed by clicking the “more” link in the Search box at the top of the page. It is intended to be a free-form field to include any extra metadata to a test that is not already supported officially by Shepherd as a flag or a field in Shepherd. It is sometimes used to classify the type of issue that may cause a test to have a Needs Work status.

Currently, there are two such classifications:

• Metadata: For tests requiring only a metadata change
• Precision: For tests that are correct in some cases, but aren't precise enough to be correct in all cases
• Incorrect: For tests that are incorrectly designed