requirements-testing: Difference between revisions
(added canoo webtest)
(added short note about feedparser.org/tests)
|Line 21:||Line 21:|
== Existing Tools and Practice ==
== Existing Tools and Practice ==
=== Canoo WebTest ===
=== Canoo WebTest ===
Revision as of 22:49, 7 December 2005
Accceptance Testing Microformat
This is a concept for documenting acceptance tests and project requirements in XHTML. The microformat should exist to encourage and facilitate the progress and uptake of Agile development methodologies across a wide spectrum of projects and organizations.
There are numerous popular approaches to requirements definition and acceptance test/use case documentation, and various automated tools to run test fixtures and report back results.
Many of the accepted formats focus on diagram notation of use cases, but a lot of the time, test plans and documentation are captured in unstructured and arbitrary styles, often in Word documents or similar. One of the major issues with working with requirements in this document format is that it is very easy to fall out of step with revisions and changes, especially once development cycles are up and running on a project. Older requirements information tends to rot and become incorrect very quickly, unless documents evolve directly with the development process itself. Word documents are hard to author collaboratively in this way, and encourage clients and developers to take an adversarial stance.
Could there be a delarative equivalent of a UML use case diagram in XHTML? What would this actually achieve? It's easy enough to do in XML, but this wouldn't be viewable directly in a document format. Good idea not to overcomplicate the simple description of a user role, and possible interactions with other roles/services.
A simple HTML vocabulary could make it much easier to share structured requirements information through a wiki type format, which could thus be updated in realtime to reflect changing requirements and new knowledge about the conditions of a system.
Automated Acceptance Testing
To make use of the full benefits of automated acceptance testing, there needs to be a way to integrate code generation and test reporting directly into a workflow by integrating with the tools that are already being used in the common office environment. FitNesse (wiki based) and Arbiter (document based) tools are two of the steps in this direction. Arbiter uses list items to define test procedures, wheras FitNesse uses tables to define expectations. Apart from that, they work in a very similar way: upload a document/wiki page -> parse the test definitions -> generate the test code -> run the test code -> report back results. But they're still too attached to the concept of a "document" as a separate editable file, rather than a dynamic web resource.
Existing Tools and Practice
Feed Parser Test Suite
The XML files from the Feed Parser Test Suite created by Mark Pilgrim are prefixed with a comment containing a description and expectation, decoupling the test assertions from imperative code:
<!-- Description: Atom namespace (official) Expect: bozo and feed['title'] == u'Example Atom' --> <!-- Description: entry contributor homepage Expect: bozo and entries['contributors']['url'] == u'http://example.com/' -->
Canoo WebTest runs test steps defined in the Ant XML format:
<target name="login" > <webtest name="normal" > &config; <steps> <invoke description="get Login Page" url="login.jsp" /> <verifyTitle description="we should see the login title" text="Login Page" /> <setInputField description="set user name" name="username" value="scott" /> <setInputField description="set password" name="password" value="tiger" /> <clickButton description="Click the submit button" label="let me in" /> <verifyTitle description="Home Page follows if login ok" text="Home Page" /> </steps> </webtest> </target>
open click clickAndWait verifyTitle verifyValue
Selenium test commands are defined using arbitrary HTML tables. This example is from the basic tests shipped with the framework, and shows how to specifiy assertions against an HTML select element:
<table cellpadding="1" cellspacing="1" border="1"> <tbody> <tr> <td rowspan="1" colspan="3">Test Select<br> </td> </tr> <tr> <td>open</td> <td>./tests/html/test_select.html</td> <td> </td> </tr> <tr> <td>select</td> <td>theSelect</td> <td>index=4</td> </tr> <tr> <td>verifySelected</td> <td>theSelect</td> <td>Fifth Option</td> </tr> </tbody> </table>
There are no specific semantic attributes used - the test tool simply extracts commands from the columns of the table.
Watir is a testing tool, written in Ruby that implements a domain specific language for driving web tests with Internet Explorer.
Example (using the browser):
ie = IE.new ie.goto("http://microformats.org") ie.contains_text("Designed for humans first")
Example (clicking a link):
Would click the hyperlink:
A list of possible commands for a a web acceptance testing tool in basic English:
- Start at [link] (Go to [link]) - Click [text][submit] - Set field [name] to [value] (Add [name] to [value) - Should see (Expect to see)
Communicating and Sharing Acceptance Test Data
Who is the audience? Clients, developers, stakeholders. What do they want to see? An itemized list of test cases - whether the tests are completed or not, and if incomplete, where did the failures occur?
Current markup ideas:
<div class="test"> <h3 class="case">Local Login Test (incomplete)</h3> <ol class="results"> <li>Start at http://localhost/login <span class="pass">(pass)</span></li> <li>Expect to see "Please enter your login" <span class="fail">(fail)</span></li> </ol> </div>
xoTest: XOXO outlines for defining macros
Example of how the compact attribute could be used to define a test "macro". In this case, "Login to localhost" becomes shorthand for referring to the sequence of nested steps.
<ol class="xoxo test"> <li>Add A Link <dl> <dt>Tests the process of adding a link</dt> <dd>additional notes here</dd> </dl> <ol> <li>Login to localhost <ol compact="compact" class="macro"> <li>Go to http://localhost/login</li> <li>Set username to "user"</li> <li>Set password to "pass"</li> <li>Click submit</li> </ol> </li> <li>Click "Add Link"</li> <li>Set url to "http://microformats.org/wiki/XOXO"</li> <li>Set description to "Extensible Open XHTML Outlines"</li> <li>Click submit</li> <li>Should see "Link was saved successfully"</li> </ol> </li> </ol>
This raises some interesting possibilities for both documentation and test generation. As well as functioning as use case documentation, this basic list could be used as input to generate a set of JWebUnit (Java) or Simpletest (PHP) test cases and assertions.