Test Suites

(Difference between revisions)

Jump to: navigation, search
(Initial µf test suite page.)
Current revision (17:18, 2 December 2012) (view source)
(update header warning)
 
(12 intermediate revisions not shown.)
Line 1: Line 1:
<entry-title>Test Suites</entry-title>
<entry-title>Test Suites</entry-title>
 +
 +
'''Note: This page reflects 2009 era test suite efforts for a set of stable microformats as of that time. See the current [[test-suite]] page for modern [[microformats2]] test suite efforts. [[User:Tantek|Tantek]] 17:18, 2 December 2012 (UTC)'''
 +
 +
----
So, you want to parse microformats in your app? Awesome. There are [[parsers]] in many languages.  
So, you want to parse microformats in your app? Awesome. There are [[parsers]] in many languages.  
A critical effort to assist in interoperability and completeness of microformat parsing is the Test Suite. This page documents the effort to produce test cases for each microformat and parsing pattern, so that you can verify the operations of your parser.
A critical effort to assist in interoperability and completeness of microformat parsing is the Test Suite. This page documents the effort to produce test cases for each microformat and parsing pattern, so that you can verify the operations of your parser.
-
 
-
'''NB''': ''This page is new. There is a limited test suite available already for [[hCard]], [[hCalendar]], [[hReview]] and [[hAtom]]. It is somewhat outdated and unmaintained, but should give you a starting point. This effort hopes to produce something far more robust and extensive, as well as setting a precedent and framework for future microformats and test suites.'''
 
-
 
-
This page will be expanded.
 
==Get the Tests==
==Get the Tests==
-
; hCard
+
specific microformats:
-
: ???
+
; [[hCard]]
-
; hCalendar
+
: http://ufxtract.com/testsuite/hcard/
-
: ???
+
: ...
-
; hAtom
+
; [[hCalendar]]
-
: ???
+
: [[value-class-date-time-tests#hCalendar|value class pattern date and time hCalendar tests]]
-
; hReview
+
: http://ufxtract.com/testsuite/hcalendar/
-
: ???
+
: ...
-
; Geo
+
; [[hReview]]
-
: ???
+
: ...
 +
; [[hAtom]]
 +
: ...
 +
; [[adr]]
 +
: ...
 +
; [[geo]]
 +
: http://ufxtract.com/testsuite/geo/default.htm
 +
: ...
 +
; [[hResume]]
 +
: http://ufxtract.com/testsuite/hresume/default.htm
 +
: ...
-
==Goals of Test Suite Project==
 
-
We need to support specifications with better tests. There's no other way for parser writers to verify the correctness and completeness of their code without them. Microformats are getting adopted by big players, and implementations are having bigger and bigger impact. It's critical that implementers don't reinvent the testing wheel for each new project.
+
patterns across microformats:
 +
; [[value-class-pattern]]
 +
: [[value-class-pattern-tests]]
 +
; [[include-pattern]]
 +
: ...
 +
 +
==Goals of Test Suite Project==
 +
We need to support specifications with better tests. There's no better way for parser writers to verify the correctness and completeness of their code without them. Microformats are getting adopted by big players, with large, stable implementations and are having bigger and bigger impact. It's critical that implementers don't reinvent the testing wheel for each new project.
-
* Verify the correctness of microformats parsing in parsers
+
We need to provide a suite to:
-
* Indicate the completeness of parser implementations
+
* Verify the correctness of microformats parsing
-
* Ensure that global parsing patterns like [[abbr-design-pattern]] and [[value-class-pattern]] are implemented accurately
+
* Indicate the completeness of a parser implementation
 +
* Ensure that global patterns like [[abbr-design-pattern]] and [[value-class-pattern]] are implemented accurately
* Provide a base set of tests for implementations to build their automated test scripts
* Provide a base set of tests for implementations to build their automated test scripts
-
* Encourage implementations to put their tests into this combined effort
+
* Encourage implementations to put their tests into this combined effort, rather than working alone
==Test Structure==
==Test Structure==
-
 
Each test takes the form of:
Each test takes the form of:
-
* Valid HTML 4 or XHTML 1.0 compatible input file
+
* Valid HTML 4 or XHTML 1.0 compatible file, containing the HTML mark-up for the test itself, along with the assertions used to validate the test.
-
* JSON output file, representating parsed data structure
+
* Assertions are described using a <code>test-fixture</code> mark-up pattern, invented by Glen Jones for his [http://www.ufxtract.com/testsuite/ µfextract tests]. ''This pattern needs to be documented here.''
-
'''JSON is the output format because:'''
+
This generic format has the following advantages:
-
* Parsers are available for most languages, and is less ambiguous for this purpose than inventing some bespoke XML format.
+
* '''It is language implementation agnostic''', so never has to be ported into other languages (e.g. PHPUnit tests ported to JUnit ported to YUITest ported to…)
-
* It's useful that all tests return the same output format. Early tests returning vcf (vcard), ics (iCal) and Atom were useful for early implementations, but parsers and consuming cases are now more sophisticated so providing raw data to verify against is more useful.
+
* '''The test case and assertions are contained in the same physical file'''. One file per test is easier to manage than multiple files associated in name.
 +
* '''The test is self-documenting'''. Because the test is written in HTML, it can contain explanation alongside the assertions, can be verified easily by hand, as well as parsed by machines.
 +
* The <code>test-fixture</code> format can be tested using itself.
 +
* Dedicated automated test suites for languages (PHP, Python, JavaScript, Ruby, etc.) can be generated from the HTML using a single script. One new test case written using <code>test-fixture</code> can create tests for all languages.
 +
* It's useful that all tests be in the same output format. Early tests for microformats returned native formats, such as <code>vcf</code> (vcard), <code>ics</code> (iCal) and Atom. These were useful in the context of early implementations, but parsers and consuming cases are now more sophisticated so providing assertions to verify against is more useful.
-
There is to be one test suite per format. In some cases, such as [[geo]] and [[adr]] with [[hCard]], the address and geo portions will be included in the geo and adr test suites and hCard parsers will be expected to include cases from all three suites.
+
'''There is to be one test suite per format'''. In some cases, such as [[geo]] and [[adr]] with [[hCard]], the address and geo portions will be included in the geo and adr test suites and hCard parsers will be expected to run cases from all three suites.
-
Each suite is organized as follows
+
Tests using global patterns, such as value-class and abbr-design-pattern should be included in tests for ''all'' vocabularies, where appropriate.
-
* <kbd>/stable</kbd> contains tests for the stable, specified aspects of a specification.
+
Each vocabulary suite is organized as follows
-
** <kbd>/stable/new</kbd> is the initial location for each test. Tests are checked into <kbd>new</kbd> first, pending peer-review. After review, the test is moved into <kbd>verified</kbd>.
+
 
-
** <kbd>/stable/verified</kbd> contains tests for stable aspects of a specification which have been sanity checked by at least one other. These tests will be presented (and should be) 100% correct and reliable.
+
* <kbd>/stable</kbd> contains tests for the stable, specified aspects of each vocabulary.
-
* <kbd>/experimental</kbd> is a repository for brainstorm tests. If a brainstorm developers to point of building an experimental implementation, the tests will be filed under <kbd>/experimental/brainstorm-name</kbd>. This keeps them separate from the stable branch, but makes it easy to merge them in if a brainstorm results in an update to the spec.
+
** <kbd>/stable/new</kbd> is the initial location for a new test. Tests are checked into <kbd>new</kbd> first, pending peer-review. After review, the test is moved into <kbd>verified</kbd>.
 +
** <kbd>/stable/verified</kbd> contains tests for stable aspects of a specification which have been sanity checked by at least one other. These tests will be presented as (and should be) 100% correct and reliable.
 +
* <kbd>/experimental</kbd> is a repository for brainstorm tests, for experimental implementations. The tests will be filed under <kbd>/experimental/brainstorm-name</kbd>. This keeps them separate from the stable branch, but makes it easy to merge them in if a brainstorm is successful and results in an update to the spec.
* <kbd>/old</kbd> contains tests that don't match the proposed test suite format/criteria. They're useful and should be ported to the new form, but are filed separately.
* <kbd>/old</kbd> contains tests that don't match the proposed test suite format/criteria. They're useful and should be ported to the new form, but are filed separately.
==Establishment Tasks==
==Establishment Tasks==
-
 
* Get the existing test cases from <code>hg.microformats.org</code> and into a Github repository.  
* Get the existing test cases from <code>hg.microformats.org</code> and into a Github repository.  
** The collaboration functionality of Github is well suited to the development of new tests.
** The collaboration functionality of Github is well suited to the development of new tests.
 +
* Get the existing <code>test-fixture</code> tests integrated
 +
** Needs Public Domain release from Glenn Jones. Done [[test-fixture]]
* Make people aware of it!
* Make people aware of it!
-
* Get feedback from other parser authors that the proposed HTML+JSON test cases are suitable.
+
* Compile documentation for <code>test-fixture</code>
-
** Document the JSON object format conventions.
+
* Fill in all the blanks on this page.
* Fill in all the blanks on this page.
* Rewrite the existing hCard, hCalendar and hAtom tests to transform into JSON rather than native formats
* Rewrite the existing hCard, hCalendar and hAtom tests to transform into JSON rather than native formats
-
* Produce a tool in JavaScript to generate the test result JSON from an HTML input sample
+
* Encourage production open source scripts to convert a <code>test-fixture</code> document into a runnable test case in major languages:
-
** Making it easier to produce new test case results
+
** PHP, Perl, Ruby, Python, .NET, Java
-
** Generate a test case from any web page using a bookmarklet
+
** Produce <code>test-fixture</code> test cases for <code>test-fixture</code> itself.
-
** Produce a reference implementation
+
 
 +
Later, thought should be given to how assertions can be used to test implied meanings in microformats. For example, we cannot test ‘hCard is an organisation’ using a simple assert, since that is not a field of hCard (but would be a data field in an implementation). Initially, these implied meanings should just be documented in the tests, and applicable implementations can take note.
 +
 
 +
== past tests ==
 +
There are also some older test suites available for [[hCard]], [[hCalendar]], [[hReview]] and [[hAtom]]. They are somewhat outdated and unmaintained, but should give you a starting point. The effort described on this page hopes to produce something far more robust and extensive, as well as setting a precedent and framework for future microformats and test suites.
 +
 
 +
==Related==
 +
* [[test-fixture]]
 +
* [[parsers|Parsers]]
 +
* [[debugging-tools|Debugging Tools]]
 +
* [http://www.ufxtract.com/testsuite/ ufXtract test suite]

Current revision


Note: This page reflects 2009 era test suite efforts for a set of stable microformats as of that time. See the current test-suite page for modern microformats2 test suite efforts. Tantek 17:18, 2 December 2012 (UTC)


So, you want to parse microformats in your app? Awesome. There are parsers in many languages.

A critical effort to assist in interoperability and completeness of microformat parsing is the Test Suite. This page documents the effort to produce test cases for each microformat and parsing pattern, so that you can verify the operations of your parser.

Contents

Get the Tests

specific microformats:

hCard
http://ufxtract.com/testsuite/hcard/
...
hCalendar
value class pattern date and time hCalendar tests
http://ufxtract.com/testsuite/hcalendar/
...
hReview
...
hAtom
...
adr
...
geo
http://ufxtract.com/testsuite/geo/default.htm
...
hResume
http://ufxtract.com/testsuite/hresume/default.htm
...


patterns across microformats:

value-class-pattern
value-class-pattern-tests
include-pattern
...

Goals of Test Suite Project

We need to support specifications with better tests. There's no better way for parser writers to verify the correctness and completeness of their code without them. Microformats are getting adopted by big players, with large, stable implementations and are having bigger and bigger impact. It's critical that implementers don't reinvent the testing wheel for each new project.

We need to provide a suite to:

Test Structure

Each test takes the form of:

This generic format has the following advantages:

There is to be one test suite per format. In some cases, such as geo and adr with hCard, the address and geo portions will be included in the geo and adr test suites and hCard parsers will be expected to run cases from all three suites.

Tests using global patterns, such as value-class and abbr-design-pattern should be included in tests for all vocabularies, where appropriate.

Each vocabulary suite is organized as follows

Establishment Tasks

Later, thought should be given to how assertions can be used to test implied meanings in microformats. For example, we cannot test ‘hCard is an organisation’ using a simple assert, since that is not a field of hCard (but would be a data field in an implementation). Initially, these implied meanings should just be documented in the tests, and applicable implementations can take note.

past tests

There are also some older test suites available for hCard, hCalendar, hReview and hAtom. They are somewhat outdated and unmaintained, but should give you a starting point. The effort described on this page hopes to produce something far more robust and extensive, as well as setting a precedent and framework for future microformats and test suites.

Related

Test Suites was last modified: Sunday, December 2nd, 2012

Views