| Last revisionBoth sides next revision |
| spec:teststatus [2012/05/03 14:15] – created dschulze | spec:teststatus [2012/05/03 16:11] – plinss |
|---|
| |
| ===== Prolog ===== | ===== Prolog ===== |
| Testing is an important part of writing specifications. It helps verifying the validity of specified features. But tests can also be used as examples for developers and serve implementers as verification tool of their own implementation. Therefore tests can be seen as the second, practical part of the specification, beside the written part. For this reasons every part of a specification must be tested. In general the intention is: as much tests as possible. See [[test:|Testing]] for more details about writing tests. | |
| |
| The preferred way to write tests for the CSS WG, SVG WG and the FX task force are the so called [[test:reftest|Writing Reftests]]. | Testing is an important part of writing specifications. It helps verifying the validity of specified features. But tests can also be used as examples for developers and serve implementers as verification tool of their own implementation. Therefore tests can be seen as the second, practical part of the specification, beside the written part. For this reasons every part of a specification must be tested. In general the intention is: every testable assertion in a specification must have at least one test. See [[test:|Testing]] for more details about writing tests. |
| | |
| | Tests for the CSS WG, SVG WG and the FX task force must be written to the [[test:format|test format guidelines]]. Where the tested features can't be easily verified in script, the preferred form of test is a [[test:reftest|reftest]]. |
| |
| ===== Introduction ===== | ===== Introduction ===== |
| | |
| To support developers as well as specification authors, specifications should have detailed information about the testing status on each specific feature. | To support developers as well as specification authors, specifications should have detailed information about the testing status on each specific feature. |
| |
| For authors a rarely tested section indicates that there might be specification issues that are not solved yet. It can also mean that this section needs more clarification. | For authors a rarely tested section indicates that there might be specification issues that are not solved yet. It can also mean that this section needs more clarification. |
| |
| For a better transparency these information should be prompted on the specification itself. And even better: To every assertable part of the specification. This article describes how to do that in the combination with the testing system [[http://test.csswg.org/shepherd/|Shepherd]] for tests submitted to the CSS test suite. | For a better transparency this information should be prompted on the specification itself. And even better: To every assertable part of the specification. This article describes how to do that in the combination with the testing system infrastructure used by the CSS working group. |
| |
| ===== Adding assertions to a specification ===== | ===== Identifying assertions within a specification ===== |
| The first step is to add assertions into a specification. At the moment this is done by adding a unique identifier to every headline on the text with the 'id' attribute. It is good practice to use the same name on the id as for the title of the section. Example: | |
| | Currently tests are mapped to the most relevant section of the specification. The CSS testing systems will soon be able to recognize testable assertions within a narrower scope. |
| | |
| | The first step is to ensure that all section headings within the specification have unique identifiers set to their 'id' attribute. It is good practice to use the same name on the id as for the title of the section. Example: |
| |
| <code> | <code> |
| </code> | </code> |
| |
| Ids helps [[http://test.csswg.org/shepherd/|Shepherd]] to recognize assertions in a specification and map submitted tests to this assertion. | Second, all testable assertions within the specification should also have a unique 'id' attribute on their smallest containing element. The paragraph or even sentence level is generally suitable. Non-testable areas of the specification must have one of the following class attributes: 'example', 'note', 'doc', 'no-toc', 'informative'. |
| |
| ===== Write tests ===== | ===== Binding tests to the specification ===== |
| Obviously [[http://test.csswg.org/shepherd/|Shepherd]] needs tests for the mapping to assertions. If not done so far, it is the best time to write tests :-). | |
| |
| ===== Creating a testing harness ===== | When writing the tests for a specification, the test must contain one or more [[test:format#links|links]] to the area(s) of the specification being tested. The fragment identifier of the link should be the 'id' of the heading or containing element of the assertion(s) under test. |
| Even if tests can be committed to the CSS test suite. It is a requirement to create a so called test harness per specification to embed information about the testing status. This test harness is a collection of all submitted tests to the CSS test suite per specification. All tests get sorted by the tested sections and subsections. | |
| |
| This test harness must be created by e.g Peter Linss (W3C). The testing harness will be synchronized with the CSS test suite once a day. Examples for existing test harnesses can be found here: [[http://test.csswg.org/harness/]]. | ===== Creating a test suite ===== |
| | |
| | Test suites are currently created manually. This will soon be a fully automated process. |
| | |
| | Once a sufficient number of tests have been submitted, [[mailto:peter.linss@hp.com|email Peter]] to have the test suite generated. At that point, the test suite will be built daily and available for testing in the [[http://test.csswg.org/harness|test harness]]. |
| |
| ===== Embedding testing status into the specification ===== | ===== Embedding testing status into the specification ===== |
| Once the test harness was created, the testing status information can be embedded into the specification. This is done by adding the following script to the header of the specification: | |
| | Once the test suite is created, the testing status information can be embedded into the specification. This is done by adding the following script to the header of the specification: |
| |
| <code> | <code> |
| <script src='http://test.csswg.org/harness/annotate.js#<specification>' type='text/javascript' defer></script> | <script src='http://test.csswg.org/harness/annotate.js#<test-suite>' type='text/javascript' defer></script> |
| </code> | </code> |
| |
| Note that '<specification>' must be replaced by the name of the specification in the test harness. This is e.g 'CSS3-TRANSFORMS_DEV' for CSS3 Transforms. | Note that '<test-suite>' must be replaced by the identifier used by the testing harness for the test suite. This can be found in the URL of the testing pages within the test harness. This is e.g 'CSS3-TRANSFORMS_DEV' for CSS3 Transforms. |
| |
| After uploading the changes, the script should run immediately. If you don't see any testing information on the specification, that either means you used the wrong name for the specification, no tests were found, or the file name of the specification is unknown. For the last case ask Peter Linss for advice. | The script will run every time the specification in viewed and will inject test information directly into the specification. Every section heading that has tests will have an annotation containing links to the test in the harness and the current result state. If you don't see any testing information on the specification, that either means you used the wrong name for the test suite, no tests were found, or the test suite has not been set up yet. For the last case [[mailto:peter.linss@hp.com|email Peter]] for advice. |
| |
