[microformats-discuss] hTest

[Mark Rickerby]

> So why not construct a little vocabulary to extend what's already in
> DOAP to describe tests? The vocabulary could be expressed as a
> microformat (so it could appear on Wikis a la Fitnesse) but would also
> be available for arbitrary machine processing via RDF.

Hi Danny,

This is something I've run into with some work on the Sourceforge
project Arbiter [1]. Arbiter is a document repository and acceptance
test generator, loosely based on the principles of FIT [2]. The basic
concept is to declare test cases as bulleted or numbered lists in a
document. The Arbiter server will parse the test cases, build the
acceptance tests in code (eg: Simpletest [3] or JWebUnit [4]) and then
run the tests, appending the results (pass/fail) back into the
document itself.

eg: an Arbiter test case would look like:

Client Log In

1) Go to http://localhost/login/
2) Set name to mark
3) Set password to something
4) Click submit
5) Expect to see "Welcome"

This would generate code that automates the process of navigating the
site, filling out forms, and making assertions about content. Running
the test would result a pass/fail notice for each line.

Our initial experiments were focused on RTF, because that provides
direct access to MS Word/Openoffice docs, which are widely used by
managers (target audience). However, some recent feedback we've had
indicates that developers especially would be more interested in using
HTML lists to mark up test cases. That has always been part of the
plan, but was intended to be factored into the project further down
the track. Getting HTML lists in there would actually be fairly
trivial at the moment - RTF is a lot more of a hassle due to
divergences in document formats, and the fact that none of us have
worked with it at all before.

I'm not totally convinced there is a pressing need for a microformat
with our input situation, mainly because the approach to parsing we
are considering is more based on the natural language  used, than
actually analyzing semantic markup. For example a line in a document
like {Should see "Arbiter"} would be detected as a test assertion
because it is 1) an item in a list, and 2) contains the assertion
"should see".

eg:

<ol>
  <li>Start at http://microformats.org</li>
  <li>Should see "Latest microformats news"</li>
</ol>

I would want this basic list format to be capable of building a test
without having to add any additional semantic attributes. I don't know
whether I am being far too ambitious here, the main thing is that I am
*really* bored of manually typing in the same PHP code over and over
again to build tests, and this would make things *much easier*.

The possibilities for a microformat become more interesting when
considering reporting results back and distributing the tests. It
would be nice to be able to feed in tests from RTF or basic HTML
lists, and render them back with more standardized markup that could
be displayed directly in the browser, and also distributed to
downstream aggregators, collected for use in separate testing tools,
or added to/scraped from wiki pages, etc.

If there was a common microformat for defining acceptance test cases,
I would definitely make sure that Arbiter supported it. I don't know
anything about DOAP though, I'm just musing on things from my
perspective as a frequent user of Simpletest.

At the moment our vocabulary is not much more than: class="pass",
class="fail"...

Regards,
Mark

[1] http://arbiter.sf.net
[2] http://fit.c2.com
[3] http://simpletest.org
[4] http://jwebunit.sourceforge.net/