Differences

This shows you the differences between two versions of the page.

Link to this comparison view

test:format [2012/04/16 13:53]
fantasai [Test Assertions]
test:format [2013/11/11 18:21] (current)
rhauck added deprecation note & redirect to testthewebforward.org
Line 1: Line 1:
 +<​html>​
 +<​strong>​
 +<div style="​color:​ red; font-size: 20px; border: 2px solid red; padding: 10px; line-height:​ 1.5; text-align: center;">​
 +This page has been deprecated and is no longer being maintained. ​
 +<​br>​For up to date information on CSS test format guidelines and test templates, see: 
 +<​br><​a href="​http://​testthewebforward.org/​docs/​test-format-guidelines.html">​http://​testthewebforward.org/​docs/​test-format-guidelines.html</​a>​
 +<​br><​a href="​http://​testthewebforward.org/​docs/​test-templates.html">​http://​testthewebforward.org/​docs/​test-templates.html</​a>​
 +</​strong>​
 +</​div>​
 +</​html>​
 +
 ====== Test Format ====== ====== Test Format ======
  
-This page describes the standard test format for CSSWG [[test:​selftest|self-describing tests]] ​and [[test:​reftest|reftests]]. The requirements are explained below; to make things easier we've also provided a template that you can copy.+This page describes the standard test format for CSSWG [[test:​selftest|self-describing tests]][[test:​reftest|reftests]] and [[test:​scripttest|script tests]]. The requirements are explained below; to make things easier we've also provided a template that you can copy.
  
-The recommended structure for CSS tests is a [[selftest|self-describing]] [[reftest|reftest]]. If it's possible for a test to be a [[reftest|reftest]],​ it must be a reftest. (We prefer reftests to also be self-describing or otherwise easy for humans to interpret, but this is not a requirement.)+The recommended structure for CSS tests is a [[selftest|self-describing]] [[reftest|reftest]]. If it's possible for a test to be a [[reftest|reftest]],​ it must be a reftest. (We prefer reftests to also be self-describing or otherwise easy for humans to interpret, but this is not a requirement). [[test:​scripttest|script tests]] should be used for scenarios such as testing a JS API or adding automation to a [[reftest|reftest]].
  
 ===== Design Requirements ===== ===== Design Requirements =====
  
-The following are requested of all tests submitted to the CSSWG, both for [[test:​reftest|reftests]] and [[test:​selftest|self-describing tests]]:+The following are requested of all tests submitted to the CSSWG, both for [[test:​reftest|reftests]], [[test:​scripttest|script tests]], ​and [[test:​selftest|self-describing tests]]:
  
   ; Short   ; Short
Line 20: Line 31:
 ===== Acceptable Test Formats ===== ===== Acceptable Test Formats =====
  
-The preferred submission format for CSSWG tests is XHTML in UTF-8. HTML is also acceptable; however note that the test source ​will be parsed and reserialized.+The preferred submission format for CSSWG tests is either ​XHTML or HTML5, ​in UTF-8. HTML < 5 is also acceptable, but it will be processed by an HTML5 compatible parser. SVG is also acceptable where necessary
  
-Images must be in PNG format.+Note that in general, the test source will be parsed and re-serialized,​ even in its source format. The re-serialization will cause minor changes to the test file, notably: attribute values will always be quoted, whitespace between attributes will be collapsed to a single space, duplicate attributes will be removed, optional closing tags will be inserted, and invalid markup will be normalized. If these changes should make the test inoperable, for example if the test is testing markup error recovery, add the [[#​requirement-flags|flag]] '​asis'​ to prevent re-serialization. This flag will also prevent format conversions so it may be necessary to provide alternate versions of the test in other formats (XHTML, HTML, etc.) 
 + 
 +Images must be in either ​PNG or SVG format. ​(PNG is preferred where raster images can be used.)
  
 A set of scripts will generate the various formats (XHTML, HTML, XHTML for Printers) from this source version. A set of scripts will generate the various formats (XHTML, HTML, XHTML for Printers) from this source version.
  
-Tests must be //​valid// ​XHTML, so please [[http://​validator.w3.org/​|validate your tests]] before submission. ​ For tests that use the HTML style header, the validator may report errors on the flags and assert metadata. ​ These specific errors can be ignored - this is a known issue and work is in progress to correct the problem.+Tests must be //valid//, so please [[http://​validator.w3.org/​|validate your tests]] before submission. ​ For tests that use the HTML style header, the validator may report errors on the flags and assert metadata. ​ These specific errors can be ignored - this is a known issue and work is in progress to correct the problem. 
 + 
 +When using any characters beyond the ASCII set, in any encoding, the character encoding must be specified properly per the specification of the source format. 
 + 
 +**If the test uses the Ahem font, make sure its computed font-size is a multiple of 5px**, otherwise baseline alignment may be rendered inconsistently (due to rounding errors introduced by certain platforms'​ font APIs). We suggest to use a minimum computed font-size of 20px. 
 +  * Eg. Bad: {font: 1in/1em Ahem;} /* computed font-size is 96px */ 
 +  * Eg. Bad: {font: 1in Ahem;} 
 +  * Eg. Bad: {font: 1em/1em Ahem} /* with computed 1em font-size being 16px */ 
 +  * Eg. Bad: {font: 1em Ahem;} /* with computed 1em font-size being 16px */ 
 +  * Eg. Good: {font: 100px/1 Ahem;} 
 +  * Eg. Good: {font: 1.25em/1 Ahem;} /* with computed 1.25em font-size being 20px */ 
 + 
 +**If the test uses the Ahem font, make sure the line-height on block elements is specified; avoid '​line-height:​ normal'​**. Also, for absolute reliability,​ the difference between computed line-height and computed font-size should be dividable by 2.  
 +  * Eg. Bad: {font: 1.25em Ahem;} /* computed line-height value is '​normal'​ */ 
 +  * Eg. Bad: {font: 20px Ahem;} /* computed line-height value is '​normal'​ */ 
 +  * Eg. Bad: {font-size: 25px; line-height:​ 50px;} /* the difference between computed line-height and computed font-size is **not** dividable by 2. */ 
 +  * Eg. Good: {font-size: 25px; line-height:​ 51px;} /* the difference between computed line-height and computed font-size is dividable by 2. */
  
 ===== Template for New Tests ===== ===== Template for New Tests =====
  
-A template for new tests follows. Copy and paste the code below into a new file or save {{test:​css2.1:​test-topic-000.xht|this template file}} and replace the capitalized ​parts as described below.+A template for new tests follows. Copy and paste the code below into a new file or save {{test:​css2.1:​test-topic-000.xht|this template file}} and replace the bracketed ​parts as described below.
  
 <code html> <code html>
Line 36: Line 65:
 <html xmlns="​http://​www.w3.org/​1999/​xhtml">​ <html xmlns="​http://​www.w3.org/​1999/​xhtml">​
  <​head>​  <​head>​
-  <​title>​CSS Test: DESCRIPTION OF TEST</​title>​ +  <​title>​CSS Test: [Title/​Scope of Test]</​title>​ 
-  <link rel="​author"​ title="​NAME_OF_AUTHOR" href="​mailto:​EMAIL OR http://​CONTACT_PAGE"/>​ +  <link rel="​author"​ title="​[Name of Author]" href="[mailto: ​or http: contact address]" /> 
-  <link rel="​help"​ href="RELEVANT_SPEC_SECTION"/>​ +  <link rel="​help"​ href="http://​www.w3.org/​TR/​[direct link to tested spec section]" /> 
-  <meta name="​flags"​ content="​TOKENS" /> +  <meta name="​flags"​ content="​[requirement flags]" /> 
-  <meta name="​assert"​ content="​TEST ASSERTION"/>+  <meta name="​assert"​ content="​Test checks that [explanation of what you're trying to test]." />
   <style type="​text/​css"><​![CDATA[   <style type="​text/​css"><​![CDATA[
-   ​CSS ​FOR TEST+   [CSS for test]
   ]]></​style>​   ]]></​style>​
  </​head>​  </​head>​
  <​body>​  <​body>​
-  ​CONTENT OF TEST+  ​<​p>​Test passes if [description of pass condition].</​p>​ 
 +  [Content of test]
  </​body>​  </​body>​
 </​html>​ </​html>​
 </​code>​ </​code>​
  
 +Alternatively,​ you can use this HTML5 template:
 +
 +<code html>
 +<​!DOCTYPE html>
 +<​title>​CSS Test: [Title/​Scope of Test]</​title>​
 +<link rel="​author"​ title="​[Name of Author]"​ href="​[mailto:​ or http: contact address]">​
 +<link rel="​help"​ href="​http://​www.w3.org/​TR/​[direct link to tested section]">​
 +<meta name="​flags"​ content="​[requirement flags]">​
 +<meta name="​assert"​ content="​Test checks that [explanation of what you're trying to test].">​
 +<​style>​
 +   [CSS for test]
 +</​style>​
 +<​body>​
 +  <​p>​Test passes if [description of pass condition].</​p>​
 +  [Content of test]
 +</​body>​
 +</​code>​
  
 ===== Template Details ===== ===== Template Details =====
Line 63: Line 110:
  
 <code html> <code html>
-<​title>​CSS Test: DESCRIPTION ​OF TEST</​title>​+<​title>​CSS Test: SCOPE OF TEST</​title>​
 </​code>​ </​code>​
  
Line 82: Line 129:
 </​note>​ </​note>​
  
 +For specifications other than CSS 2.1, you can include the module name somewhere before the colon, like "CSS Selectors Test:" or "CSS Test (Selectors):"​. ​ Do not include the module version number, since the test might get reused for the next version.
  
 ==== Credits ==== ==== Credits ====
Line 143: Line 191:
 </​code>​ </​code>​
  
-The specification link elements provide a way to align test with information in the CSS 2.1 specification.+The specification link elements provide a way to align the test with information in the specification ​being tested.
  
-  * Links should link to relevant sections within the CSS 2.1 Specification +  * Links should link to relevant sections within the specification 
-  * Use the anchors from the CSS 2.1 Specification [[http://​www.w3.org/​TR/​CSS21/#​toc|Table of Contents]]+  * Use the anchors from the specification'​s ​Table of Contents
   * A test can have multiple specification links   * A test can have multiple specification links
     * Always list the primary section that is being tested as the first item in the list of specification links     * Always list the primary section that is being tested as the first item in the list of specification links
Line 178: Line 226:
 </​code>​ </​code>​
  
-The reference link elements are used in [[.:​css2.1:​test:​reftest|reftests]] and provide the list of reference file(s) that the test should be compared to.+The reference link elements are used in [[reftest|reftests]] and provide the list of reference file(s) that the test should be compared to.
  
   * ''​match''​ references must be files that render identically to the test, but use an alternate means to do so   * ''​match''​ references must be files that render identically to the test, but use an alternate means to do so
   * Multiple ''​match''​ references are used when the test can match **any** of the reference files   * Multiple ''​match''​ references are used when the test can match **any** of the reference files
     * If a test requires multiple ''​match''​ references that all need to match (for example, to catch when a reference fails in the same way the test does), then chain the references together, i.e.: place reference links to the additional ''​match''​ references in the reference files. It is recommended that the chained reference files form a loop (e.g.: a -> b -> c -> a) so that a test linking to any reference in the chain will find all the references.     * If a test requires multiple ''​match''​ references that all need to match (for example, to catch when a reference fails in the same way the test does), then chain the references together, i.e.: place reference links to the additional ''​match''​ references in the reference files. It is recommended that the chained reference files form a loop (e.g.: a -> b -> c -> a) so that a test linking to any reference in the chain will find all the references.
-  * ''​mismatch''​ references are files that render differently than the test file. A test may have any number of ''​mismatch''​ references.+  * ''​mismatch''​ references are files that render differently than the test file. A test may have any number of ''​mismatch''​ references. The test is considered to fail if it renders the same as **any** of the ''​mismatch''​ references. 
 +    * Note that reference files may themselves have ''​mismatch''​ references. In that case the reference file must not render the same as any of its ''​mismatch''​ references in order to be considered valid. If a reference is considered invalid (by the fact of not matching any of its ''​match''​ references, or matching any of its ''​mismatch''​ references),​ then a test that refers to the reference will be considered to have failed
   * Reference files may be dedicated reference files, images, or other tests   * Reference files may be dedicated reference files, images, or other tests
  
Line 210: Line 259:
 </​code>​ </​code>​
  
-Flags document the test’s prerequisites,​ for example, the Ahem font, or a paged presentation. Multiple tokens are space separated.+Flags document the test’s prerequisites,​ for example, the Ahem font, or a paged presentation. Multiple tokens are space separated. Flag tokens are case sensitive.
  
 All flags documented elsewhere are deprecated except the following: All flags documented elsewhere are deprecated except the following:
Line 216: Line 265:
 |  ahem  | Requires the [[http://​www.w3.org/​Style/​CSS/​Test/​Ahem/​|Ahem font]] | |  ahem  | Requires the [[http://​www.w3.org/​Style/​CSS/​Test/​Ahem/​|Ahem font]] |
 |  animated ​ | Test is animated in final state. (Cannot be verified using reftests/​screenshots.) | |  animated ​ | Test is animated in final state. (Cannot be verified using reftests/​screenshots.) |
 +|  asis  | The test has particular markup formatting requirements and cannot be re-serialized. |
 |  combo  | Test, which must have an unsuffixed filename number, is strictly the union of all the suffixed tests with the same name and number. (See File name format, below.) | |  combo  | Test, which must have an unsuffixed filename number, is strictly the union of all the suffixed tests with the same name and number. (See File name format, below.) |
 |  dom  | Requires support for JavaScript and the Document Object Model (DOM) | |  dom  | Requires support for JavaScript and the Document Object Model (DOM) |
 |  font  | Requires a specific font to be installed. (Details must be provided and/or the font linked to in the test description) | |  font  | Requires a specific font to be installed. (Details must be provided and/or the font linked to in the test description) |
 |  history ​ | User agent session history is required. Testing '':​visited''​ is a good example where this may be used. | |  history ​ | User agent session history is required. Testing '':​visited''​ is a good example where this may be used. |
-|  http  | Requires HTTP headers | 
 |  HTMLonly ​ | Test case is only valid for HTML | |  HTMLonly ​ | Test case is only valid for HTML |
 +|  http  | Requires HTTP headers |
 |  image  | Requires support for bitmap graphics and the graphic to load | |  image  | Requires support for bitmap graphics and the graphic to load |
 |  interact ​ | Requires human interaction (such as for testing scrolling behavior) | |  interact ​ | Requires human interaction (such as for testing scrolling behavior) |
 |  invalid ​ | Tests handling of invalid CSS. Note: This case contains CSS properties and syntax that may not validate. | |  invalid ​ | Tests handling of invalid CSS. Note: This case contains CSS properties and syntax that may not validate. |
 +|  may  | Behavior tested is //​preferred//​ but OPTIONAL. %%[%%[[http://​www.ietf.org/​rfc/​rfc2119.txt|RFC2119]]] (These tests must be reviewed by a test suite owner or peer.) |
 |  namespace ​ | Requires support for XML Namespaces | |  namespace ​ | Requires support for XML Namespaces |
 |  nonHTML ​ | Test case is only valid for formats besides HTML (e.g. XHTML or arbitrary XML) | |  nonHTML ​ | Test case is only valid for formats besides HTML (e.g. XHTML or arbitrary XML) |
-|  may  | Behavior tested is //​preferred//​ but OPTIONAL. %%[%%[[http://​www.ietf.org/​rfc/​rfc2119.txt|RFC2119]]] (These tests must be reviewed by a test suite owner or peer.) | 
 |  paged  | Only valid for paged media | |  paged  | Only valid for paged media |
-|  should ​ | Behavior tested is RECOMMENDED,​ but not REQUIRED. %%[%%[[http://​www.ietf.org/​rfc/​rfc2119.txt|RFC2119]]] | 
 |  scroll ​ | Only valid for continuous (scrolling) media | |  scroll ​ | Only valid for continuous (scrolling) media |
 +|  should ​ | Behavior tested is RECOMMENDED,​ but not REQUIRED. %%[%%[[http://​www.ietf.org/​rfc/​rfc2119.txt|RFC2119]]] |
 +|  speech ​ | Device supports audio output. Text-to-speech (TTS) engine installed | 
 |  svg  | Requires support for vector graphics (SVG) | |  svg  | Requires support for vector graphics (SVG) |
 |  userstyle ​ | Requires a user style sheet to be set | |  userstyle ​ | Requires a user style sheet to be set |
Line 243: Line 294:
 Example 2 (multiple tokens apply): Example 2 (multiple tokens apply):
 <code html> <code html>
-<meta name="​flags"​ content="​ahem ​scroll ​image" />+<meta name="​flags"​ content="​ahem image scroll" />
 </​code>​ </​code>​
  
Line 276: Line 327:
 <​note>​ <​note>​
 Examples of good test assertions: Examples of good test assertions:
-  * "background image with no intrinsic size covers the entire padding box."​ +  * "This test checks that a background image with no intrinsic size covers the entire padding box."​ 
-  * "Word-spacing affects each space (U+0020) and non-breaking space (U+00A0)."​ +  * "This test checks that 'word-spacing' ​affects each space (U+0020) and non-breaking space (U+00A0)."​ 
-  * "If '​top'​ and '​bottom'​ are specified on an absolutely-positioned replaced element, then any remaining space is split amongst the '​auto'​ vertical margins."​ +  * "This test checks that if '​top'​ and '​bottom' ​offsets ​are specified on an absolutely-positioned replaced element, then any remaining space is split amongst the '​auto'​ vertical margins."​ 
-  * "​text-indent affects only the first line of a block container if that line is also the first formatted line of an element+  * "This test checks that 'text-indent' ​affects only the first line of a block container if that line is also the first formatted line of an element."
 </​note>​ </​note>​
  
Line 290: Line 341:
 </​code>​ </​code>​
  
-When creating styles primarily use ID or Class selectors. Inline styles should not be used unless the case is specifically testing this scenario.+When creating styles primarily use type, ID or Class selectors. Inline styles should not be used unless the case is specifically testing this scenario. Other selector types should also be avoided unless specifically testing those scenarios.
  
  
Line 309: Line 360:
  
 <code html> <code html>
-<​body>​ + <​body>​ 
- CONTENT OF TEST +   <​p>​Test passes if [description of pass condition].</​p>​ 
-</​body>​+   ​[Content of test] 
 + </​body>​
 </​code>​ </​code>​
  
Line 319: Line 371:
   * Test ''​table''​ features with both HTML table elements and ''<​div>''​ + ''​display:​ table-*''​   * Test ''​table''​ features with both HTML table elements and ''<​div>''​ + ''​display:​ table-*''​
   * Do not use the ''​style''​ attribute (inline styles) unless specifically testing that attribute   * Do not use the ''​style''​ attribute (inline styles) unless specifically testing that attribute
-  * Avoid making assumptions that are not in the [[http://www.w3.org/Style/CSS/​Test/​CSS2.1/current/​|"​most of the test suite makes the following assumptions"​]] list and document any that you do.+  * Avoid making assumptions that are not in the [[http://test.csswg.org/suites/css2.1/latest/#common|"most of the test suite makes the following assumptions"​]] list and document any that you do.
   * It's helpful to people trying to understand the test if you use meaningful class and ID names and escape invisible characters like no-break-spaces.   * It's helpful to people trying to understand the test if you use meaningful class and ID names and escape invisible characters like no-break-spaces.
-  * Indent nicely!+  * Indent nicely! ​ ​There'​s no standard for how many tabs/spaces to use; just be consistent within a file. 
 +  * In HTML5 tests, do not omit attribute quotation marks or closing tags, even when it's valid to do so.
  
 Guidelines on designing the content of the test can be found in the [[http://​www.w3.org/​Style/​CSS/​Test/​guidelines.html|CSS2.1 Test Case Authoring Guidelines]]. Guidelines on designing the content of the test can be found in the [[http://​www.w3.org/​Style/​CSS/​Test/​guidelines.html|CSS2.1 Test Case Authoring Guidelines]].
Line 331: Line 384:
  
 The file name is no longer restricted to 31 characters, but please try to keep them short. The file name is no longer restricted to 31 characters, but please try to keep them short.
 +
 +The file name should not use the underscore ("​_"​) character; please use the hyphen ("​-"​) character instead.
  
 **test-topic**\\ **test-topic**\\
Line 387: Line 442:
   * a 4-part picture   * a 4-part picture
  
-Additional generic files can be added as necessary, and should have a descriptive file name. Test-specific files should be named after the test (or test-topic if they are shared across several tests within a series). If possible tests should not rely on transparency in images, as they are converted to JPEG (which does not support transparency) for the xhtml1print version.+Additional generic files can be added as necessary, and should have a descriptive file name. Just like other file name, support files' file name should not use the underscore ("​_"​) character; use the hyphen ("​-"​) character instead. Test-specific files should be named after the test (or test-topic if they are shared across several tests within a series). If possible tests should not rely on transparency in images, as they are converted to JPEG (which does not support transparency) for the xhtml1print version.
  
  
 
test/format.txt · Last modified: 2013/11/11 18:21 by rhauck
Recent changes RSS feed Valid XHTML 1.0 Valid CSS Driven by DokuWiki