test-suite-2009

<entry-title>Test Suites</entry-title>

So, you want to parse microformats in your app? Awesome. There are parsers in many languages.

A critical effort to assist in interoperability and completeness of microformat parsing is the Test Suite. This page documents the effort to produce test cases for each microformat and parsing pattern, so that you can verify the operations of your parser.

NB: This page is new. There is a limited test suite available already for hCard, hCalendar, hReview and hAtom. It is somewhat outdated and unmaintained, but should give you a starting point. This effort hopes to produce something far more robust and extensive, as well as setting a precedent and framework for future microformats and test suites.

This page will be expanded.

Get the Tests

hCard

???

hCalendar

???

hAtom

???

hReview

???

Geo

???

Goals of Test Suite Project

We need to support specifications with better tests. There's no better way for parser writers to verify the correctness and completeness of their code without them. Microformats are getting adopted by big players, with large, stable implementations and are having bigger and bigger impact. It's critical that implementers don't reinvent the testing wheel for each new project.

We need to provide a suite to:

• Verify the correctness of microformats parsing
• Indicate the completeness of a parser implementation
• Ensure that global patterns like abbr-design-pattern and value-class-pattern are implemented accurately
• Provide a base set of tests for implementations to build their automated test scripts
• Encourage implementations to put their tests into this combined effort, rather than working alone

Test Structure

Each test takes the form of:

• Valid HTML 4 or XHTML 1.0 compatible file, containing the HTML mark-up for the test itself, along with the assertions used to validate the test.
• Assertions are described using a test-fixture mark-up pattern, invented by Glen Jones for his µfextract tests. This pattern needs to be documented here.

This generic format has the following advantages:

• It is language implementation agnostic, so never has to be ported into other languages (e.g. PHPUnit tests ported to JUnit ported to YUITest ported to…)
• The test case and assertions are contained in the same physical file. One file per test is easier to manage than multiple files associated in name.
• The test is self-documenting. Because the test is written in HTML, it can contain explanation alongside the assertions, can be verified easily by hand, as well as parsed by machines.
• The test-fixture format can be tested using itself.
• Dedicated automated test suites for languages (PHP, Python, JavaScript, Ruby, etc.) can be generated from the HTML using a single script. One new test case written using test-fixture can create tests for all languages.
• It's useful that all tests be in the same output format. Early tests for microformats returned native formats, such as vcf (vcard), ics (iCal) and Atom. These were useful in the context of early implementations, but parsers and consuming cases are now more sophisticated so providing assertions to verify against is more useful.

There is to be one test suite per format. In some cases, such as geo and adr with hCard, the address and geo portions will be included in the geo and adr test suites and hCard parsers will be expected to run cases from all three suites.

Tests using global patterns, such as value-class and abbr-design-pattern should be included in tests for all vocabularies, where appropriate.

Each vocabulary suite is organized as follows

• /stable contains tests for the stable, specified aspects of each vocabulary.
  • /stable/new is the initial location for a new test. Tests are checked into new first, pending peer-review. After review, the test is moved into verified.
  • /stable/verified contains tests for stable aspects of a specification which have been sanity checked by at least one other. These tests will be presented as (and should be) 100% correct and reliable.
• /experimental is a repository for brainstorm tests, for experimental implementations. The tests will be filed under /experimental/brainstorm-name. This keeps them separate from the stable branch, but makes it easy to merge them in if a brainstorm is successful and results in an update to the spec.
• /old contains tests that don't match the proposed test suite format/criteria. They're useful and should be ported to the new form, but are filed separately.

Establishment Tasks

• Get the existing test cases from hg.microformats.org and into a Github repository.
  • The collaboration functionality of Github is well suited to the development of new tests.
• Get the existing test-fixture tests integrated
  • Needs Public Domain release from Glenn Jones. Done test-fixture
• Make people aware of it!
• Compile documentation for test-fixture
• Fill in all the blanks on this page.
• Rewrite the existing hCard, hCalendar and hAtom tests to transform into JSON rather than native formats
• Encourage production open source scripts to convert a test-fixture document into a runnable test case in major languages:
  • PHP, Perl, Ruby, Python, .NET, Java
  • Produce test-fixture test cases for test-fixture itself.

Later, thought should be given to how assertions can be used to test implied meanings in microformats. For example, we cannot test ‘hCard is an organisation’ using a simple assert, since that is not a field of hCard (but would be a data field in an implementation). Initially, these implied meanings should just be documented in the tests, and applicable implementations can take note.

Related

• test-fixture
• Parsers
• Debugging Tools
• ufXtract test suite