| Both sides previous revisionPrevious revisionNext revision | Previous revisionLast revisionBoth sides next revision |
| test:review-shepherd [2013/07/31 16:59] – Documentation for the Shepherd review process workflow rhauck | test:review-shepherd [2013/08/01 08:57] – [Process for New Tests] rhauck |
|---|
| |
| Whether using this process or Github, test authors and reviewers must adhere to the [[https://github.com/w3c/testtwf-website/blob/gh-pages/docs/review-process.md| W3C Test Review Policy]]. FIXME | Whether using this process or Github, test authors and reviewers must adhere to the [[https://github.com/w3c/testtwf-website/blob/gh-pages/docs/review-process.md| W3C Test Review Policy]]. FIXME |
| | |
| | The general process looks like this: |
| | |
| | {{:test:review-flowchart-v2.png|}} |
| |
| ==== Process for New Tests ==== | ==== Process for New Tests ==== |
| - When a test author is ready for tests to be reviewed, s/he must push them to the repository in a directory named ''submitted''. The tests will then be present in Shepherd as [[http://test.csswg.org/shepherd/search/status/submitted|Submitted for Review]]. | - When a test author is ready for tests to be reviewed, s/he must push them to the repository in a directory named ''submitted''. The tests will then be present in Shepherd as [[http://test.csswg.org/shepherd/search/status/submitted|Submitted for Review]]. |
| - The test author should then ask for a review on the <public-css-testsuite@w3.org> mailing list. | - The test author should then ask for a review on the <public-css-testsuite@w3.org> mailing list. |
| - When a reviewer comes forward, s/he should examine the tests for correctness with respect to the [[https://github.com/w3c/testtwf-website/blob/gh-pages/docs/test-format-guidelines.md|format]] FIXME and [[https://github.com/w3c/testtwf-website/blob/gh-pages/docs/test-style-guidelines.md|style]] FIXME guidelines. | - When a reviewer comes forward, s/he should examine the tests for correctness with respect to the [[https://github.com/w3c/testtwf-website/blob/gh-pages/docs/test-format-guidelines.md|format]] FIXME and [[https://github.com/w3c/testtwf-website/blob/gh-pages/docs/test-style-guidelines.md|style]] FIXME guidelines. A [[https://github.com/w3c/testtwf-website/blob/gh-pages/docs/review-checklist.md|short checklist]] FIXME is available to assist the reviewer and a more detailed [[test:css2.1:review-checklist|CSS-specific checklist]] is also available. |
| - The reviewer should then notify the mailing list of the test review. It is not necessary to include all of the review feedback in the mail, just a link to the test suite in Shepherd. | - The reviewer should then notify the mailing list of the test review. It is not necessary to include all of the review feedback in the mail, just a link to the test suite in Shepherd. |
| - **If the test passes review:** Go to the next step. | - **If the test passes review:** Go to the next step. |
| - When the updated tests are submitted, another notification should be sent to the mailing list asking for review. | - When the updated tests are submitted, another notification should be sent to the mailing list asking for review. |
| - A reviewer looks at the changes and decides if they satisfy the original review feedback. Likewise, ideally this is original reviewer, but may also be any other qualified person. | - A reviewer looks at the changes and decides if they satisfy the original review feedback. Likewise, ideally this is original reviewer, but may also be any other qualified person. |
| - Once the test/changes has pass review, the reviewer should [[https://github.com/w3c/testtwf-website/blob/gh-pages/docs/test-templates.md#reviewer| note their acceptance in the test]] FIXME and re-submit the files to Mercurial. The commit message should indicating the test passed review and will be attached to the test in Shepherd. The test status in Shepherd will then change to [[ http://test.csswg.org/shepherd/search/status/accepted|Accepted]] | - Once the test/changes has pass review, the reviewer should [[https://github.com/w3c/testtwf-website/blob/gh-pages/docs/test-templates.md#reviewer| note their acceptance in the test]] FIXME and re-submit the files to Mercurial. The commit message should indicate the test passed review and will be attached to the test in Shepherd. The test status in Shepherd will then change to [[ http://test.csswg.org/shepherd/search/status/accepted|Accepted]] |
| - The [[https://test.csswg.org/shepherd/administration/testsuite/search/status/approved|Owner]] FIXME: //[This link should be exposed to all roles in Shepherd. Right now, only admins can see it]// of that suite can either approve the reviewer's judgement if the reviewer is known to be competent in this area, or review the test himself. Once the Owner is satisfied the test can be set to [[http://test.csswg.org/shepherd/search/status/approved|Approved]] in Shepherd. (If the initial reviewer was an Owner then this step is automatic). | - The [[https://test.csswg.org/shepherd/administration/testsuite/search/status/approved|Owner]] FIXME: //[This link should be exposed to all roles in Shepherd. Right now, only admins can see it]// of that suite can either approve the reviewer's judgement if the reviewer is known to be competent in this area, or review the test himself. Once the Owner is satisfied the test can be set to [[http://test.csswg.org/shepherd/search/status/approved|Approved]] in Shepherd. (If the initial reviewer was an Owner then this step is automatic). |
| |
| * [[http://test.csswg.org/shepherd/search/spec/css21/status/issue/whiteboard/Precision|Precision]]: For tests that are correct in some cases, but aren't precise enough to be correct in all cases | * [[http://test.csswg.org/shepherd/search/spec/css21/status/issue/whiteboard/Precision|Precision]]: For tests that are correct in some cases, but aren't precise enough to be correct in all cases |
| * [[http://test.csswg.org/shepherd/search/spec/css21/status/issue/whiteboard/Incorrect|Incorrect]]: For tests that are incorrectly designed | * [[http://test.csswg.org/shepherd/search/spec/css21/status/issue/whiteboard/Incorrect|Incorrect]]: For tests that are incorrectly designed |
| |
