test-suite-2009: Difference between revisions

From Microformats Wiki
Jump to navigation Jump to search
(Added related links + link to ufxtract suite)
(Updated wrt adopting the µfxtract test-fixture format)
Line 24: Line 24:
==Goals of Test Suite Project==
==Goals of Test Suite Project==


We need to support specifications with better tests. There's no other way for parser writers to verify the correctness and completeness of their code without them. Microformats are getting adopted by big players, and implementations are having bigger and bigger impact. It's critical that implementers don't reinvent the testing wheel for each new project.
We need to support specifications with better tests. There's no better way for parser writers to verify the correctness and completeness of their code without them. Microformats are getting adopted by big players, with large, stable implementations and are having bigger and bigger impact. It's critical that implementers don't reinvent the testing wheel for each new project.


* Verify the correctness of microformats parsing in parsers
We need to provide a suite to:
* Indicate the completeness of parser implementations
 
* Ensure that global parsing patterns like [[abbr-design-pattern]] and [[value-class-pattern]] are implemented accurately
* Verify the correctness of microformats parsing
* Indicate the completeness of a parser implementation
* Ensure that global patterns like [[abbr-design-pattern]] and [[value-class-pattern]] are implemented accurately
* Provide a base set of tests for implementations to build their automated test scripts
* Provide a base set of tests for implementations to build their automated test scripts
* Encourage implementations to put their tests into this combined effort
* Encourage implementations to put their tests into this combined effort, rather than working alone


==Test Structure==
==Test Structure==
Line 36: Line 38:
Each test takes the form of:
Each test takes the form of:


* Valid HTML 4 or XHTML 1.0 compatible input file
* Valid HTML 4 or XHTML 1.0 compatible file, containing the HTML mark-up for the test itself, along with the assertions used to validate the test.
* JSON output file, representating parsed data structure
* Assertions are described using a <code>test-fixture</code> mark-up pattern, invented by Glen Jones for his [http://www.ufxtract.com/testsuite/µfextract tests]. ''This pattern needs to be documented here.''


'''JSON is the output format because:'''
This generic format has the following advantages:


* Parsers are available for most languages, and is less ambiguous for this purpose than inventing some bespoke XML format.
* '''It is language implementation agnostic''', so never has to be ported into other languages (e.g. PHPUnit tests ported to JUnit ported to YUITest ported to…)
* It's useful that all tests return the same output format. Early tests returning vcf (vcard), ics (iCal) and Atom were useful for early implementations, but parsers and consuming cases are now more sophisticated so providing raw data to verify against is more useful.
* '''The test case and assertions are contained in the same physical file'''. One file per test is easier to manage than multiple files associated in name.
* '''The test is self-documenting'''. Because the test is written in HTML, it can contain explanation alongside the assertions, can be verified easily by hand, as well as parsed by machines.
* The <code>test-fixture</code> format can be tested using itself.
* Dedicated automated test suites for languages (PHP, Python, JavaScript, Ruby, etc.) can be generated from the HTML using a single script. One new test case written using <code>test-fixture</code> can create tests for all languages.
* It's useful that all tests be in the same output format. Early tests for microformats returned native formats, such as <code>vcf</code> (vcard), <code>ics</code> (iCal) and Atom. These were useful in the context of early implementations, but parsers and consuming cases are now more sophisticated so providing assertions to verify against is more useful.


There is to be one test suite per format. In some cases, such as [[geo]] and [[adr]] with [[hCard]], the address and geo portions will be included in the geo and adr test suites and hCard parsers will be expected to include cases from all three suites.
'''There is to be one test suite per format'''. In some cases, such as [[geo]] and [[adr]] with [[hCard]], the address and geo portions will be included in the geo and adr test suites and hCard parsers will be expected to run cases from all three suites.


Each suite is organized as follows
Tests using global patterns, such as value-class and abbr-design-pattern should be included in tests for ''all'' vocabularies, where appropriate.


* <kbd>/stable</kbd> contains tests for the stable, specified aspects of a specification.
Each vocabulary suite is organized as follows
** <kbd>/stable/new</kbd> is the initial location for each test. Tests are checked into <kbd>new</kbd> first, pending peer-review. After review, the test is moved into <kbd>verified</kbd>.
 
** <kbd>/stable/verified</kbd> contains tests for stable aspects of a specification which have been sanity checked by at least one other. These tests will be presented (and should be) 100% correct and reliable.
* <kbd>/stable</kbd> contains tests for the stable, specified aspects of each vocabulary.
* <kbd>/experimental</kbd> is a repository for brainstorm tests. If a brainstorm developers to point of building an experimental implementation, the tests will be filed under <kbd>/experimental/brainstorm-name</kbd>. This keeps them separate from the stable branch, but makes it easy to merge them in if a brainstorm results in an update to the spec.
** <kbd>/stable/new</kbd> is the initial location for a new test. Tests are checked into <kbd>new</kbd> first, pending peer-review. After review, the test is moved into <kbd>verified</kbd>.
** <kbd>/stable/verified</kbd> contains tests for stable aspects of a specification which have been sanity checked by at least one other. These tests will be presented as (and should be) 100% correct and reliable.
* <kbd>/experimental</kbd> is a repository for brainstorm tests, for experimental implementations. The tests will be filed under <kbd>/experimental/brainstorm-name</kbd>. This keeps them separate from the stable branch, but makes it easy to merge them in if a brainstorm is successful and results in an update to the spec.
* <kbd>/old</kbd> contains tests that don't match the proposed test suite format/criteria. They're useful and should be ported to the new form, but are filed separately.
* <kbd>/old</kbd> contains tests that don't match the proposed test suite format/criteria. They're useful and should be ported to the new form, but are filed separately.


Line 58: Line 66:
* Get the existing test cases from <code>hg.microformats.org</code> and into a Github repository.  
* Get the existing test cases from <code>hg.microformats.org</code> and into a Github repository.  
** The collaboration functionality of Github is well suited to the development of new tests.
** The collaboration functionality of Github is well suited to the development of new tests.
* Get the existing <code>test-fixture</code> tests integrated
** Needs Public Domain release from Glen Jones.
* Make people aware of it!
* Make people aware of it!
* Get feedback from other parser authors that the proposed HTML+JSON test cases are suitable.
* Compile documentation for <code>test-fixture</code>
** Document the JSON object format conventions.
* Fill in all the blanks on this page.
* Fill in all the blanks on this page.
* Rewrite the existing hCard, hCalendar and hAtom tests to transform into JSON rather than native formats
* Rewrite the existing hCard, hCalendar and hAtom tests to transform into JSON rather than native formats
* Produce a tool in JavaScript to generate the test result JSON from an HTML input sample
* Encourage production open source scripts to convert a <code>test-fixture</code> document into a runnable test case in major languages:
** Making it easier to produce new test case results
** PHP, Perl, Ruby, Python, .NET, Java
** Generate a test case from any web page using a bookmarklet
** Produce <code>test-fixture</code> test cases for <code>test-fixture</code> itself.
** Produce a reference implementation
 
Later, thought should be given to how assertions can be used to test implied meanings in microformats. For example, we cannot test ‘hCard is an organisation’ using a simple assert, since that is not a field of hCard (but would be a data field in an implementation). Initially, these implied meanings should just be documented in the tests, and applicable implementations can take note.


==Related==
==Related==

Revision as of 22:52, 19 June 2009

<entry-title>Test Suites</entry-title>

So, you want to parse microformats in your app? Awesome. There are parsers in many languages.

A critical effort to assist in interoperability and completeness of microformat parsing is the Test Suite. This page documents the effort to produce test cases for each microformat and parsing pattern, so that you can verify the operations of your parser.

'NB: This page is new. There is a limited test suite available already for hCard, hCalendar, hReview and hAtom. It is somewhat outdated and unmaintained, but should give you a starting point. This effort hopes to produce something far more robust and extensive, as well as setting a precedent and framework for future microformats and test suites.

This page will be expanded.

Get the Tests

hCard
???
hCalendar
???
hAtom
???
hReview
???
Geo
???

Goals of Test Suite Project

We need to support specifications with better tests. There's no better way for parser writers to verify the correctness and completeness of their code without them. Microformats are getting adopted by big players, with large, stable implementations and are having bigger and bigger impact. It's critical that implementers don't reinvent the testing wheel for each new project.

We need to provide a suite to:

  • Verify the correctness of microformats parsing
  • Indicate the completeness of a parser implementation
  • Ensure that global patterns like abbr-design-pattern and value-class-pattern are implemented accurately
  • Provide a base set of tests for implementations to build their automated test scripts
  • Encourage implementations to put their tests into this combined effort, rather than working alone

Test Structure

Each test takes the form of:

  • Valid HTML 4 or XHTML 1.0 compatible file, containing the HTML mark-up for the test itself, along with the assertions used to validate the test.
  • Assertions are described using a test-fixture mark-up pattern, invented by Glen Jones for his tests. This pattern needs to be documented here.

This generic format has the following advantages:

  • It is language implementation agnostic, so never has to be ported into other languages (e.g. PHPUnit tests ported to JUnit ported to YUITest ported to…)
  • The test case and assertions are contained in the same physical file. One file per test is easier to manage than multiple files associated in name.
  • The test is self-documenting. Because the test is written in HTML, it can contain explanation alongside the assertions, can be verified easily by hand, as well as parsed by machines.
  • The test-fixture format can be tested using itself.
  • Dedicated automated test suites for languages (PHP, Python, JavaScript, Ruby, etc.) can be generated from the HTML using a single script. One new test case written using test-fixture can create tests for all languages.
  • It's useful that all tests be in the same output format. Early tests for microformats returned native formats, such as vcf (vcard), ics (iCal) and Atom. These were useful in the context of early implementations, but parsers and consuming cases are now more sophisticated so providing assertions to verify against is more useful.

There is to be one test suite per format. In some cases, such as geo and adr with hCard, the address and geo portions will be included in the geo and adr test suites and hCard parsers will be expected to run cases from all three suites.

Tests using global patterns, such as value-class and abbr-design-pattern should be included in tests for all vocabularies, where appropriate.

Each vocabulary suite is organized as follows

  • /stable contains tests for the stable, specified aspects of each vocabulary.
    • /stable/new is the initial location for a new test. Tests are checked into new first, pending peer-review. After review, the test is moved into verified.
    • /stable/verified contains tests for stable aspects of a specification which have been sanity checked by at least one other. These tests will be presented as (and should be) 100% correct and reliable.
  • /experimental is a repository for brainstorm tests, for experimental implementations. The tests will be filed under /experimental/brainstorm-name. This keeps them separate from the stable branch, but makes it easy to merge them in if a brainstorm is successful and results in an update to the spec.
  • /old contains tests that don't match the proposed test suite format/criteria. They're useful and should be ported to the new form, but are filed separately.

Establishment Tasks

  • Get the existing test cases from hg.microformats.org and into a Github repository.
    • The collaboration functionality of Github is well suited to the development of new tests.
  • Get the existing test-fixture tests integrated
    • Needs Public Domain release from Glen Jones.
  • Make people aware of it!
  • Compile documentation for test-fixture
  • Fill in all the blanks on this page.
  • Rewrite the existing hCard, hCalendar and hAtom tests to transform into JSON rather than native formats
  • Encourage production open source scripts to convert a test-fixture document into a runnable test case in major languages:
    • PHP, Perl, Ruby, Python, .NET, Java
    • Produce test-fixture test cases for test-fixture itself.

Later, thought should be given to how assertions can be used to test implied meanings in microformats. For example, we cannot test ‘hCard is an organisation’ using a simple assert, since that is not a field of hCard (but would be a data field in an implementation). Initially, these implied meanings should just be documented in the tests, and applicable implementations can take note.

Related