machine-data

Machine Data in Microformats

Microformats are designed to mark-up human consumable information, as commonly found in the wild. But, in a number of exceptional cases it has been necessary to specify precise data formats for particular properties. Formats for dates, times and locations are standardised in a way that doesn't always match the way information is visibly published. This is necessary to make the data understandable to parsers. Similarly, there are keywords in hCard that must be written in English (telephone ‘type’ in hCard, for example).

It is necessary for these data formats to be fixed to make the data parsable by machines; the cost for a parser to support every commonly published date-time format in the world (include approximations like ‘five minutes ago’) is too high, as is handling international translation (such as mobile telephones; US-English ‘cell’ published as British English ‘mobile’).

In some cases, the human version of the data can be semantically described as an abbreviated form of the machine data, and the machine data may also be human consumable. For example, the date-design-pattern uses HTML's abbr element to expand one human date representation into the ISO 8601 form date: ‘January 1st’ is an abbreviated form of ‘2008-01-01’. The latter is also legible to humans (and can be exposed to them through tool-tips and assistive screen readers).

In other cases, this machine data is not legible to humans. In hAudio, the duration property uses ISO 8601, resulting in machine data of PT3M23S; not understandable to humans, and therefore not a valid expansion of ‘three minutes and twenty-three seconds’.

Cases of Fixed Data Formats in Microformats

The following are all current uses of fixed format machine data required by the various microformats.

hCalendar

• Uses ISO 8601 for dtstart, dtend, duration and rdate

hCard

• Telephone type keywords: voice, home, msg, work, pref, fax, cell, video, pager, bbs, modem, car, isdn, pcs.
• Address type keywords: INTL, POSTAL, PARCEL, WORK, dom, home, pref.
• Email type keywords: INTERNET, x400, pref.
• Uses ISO 8601 date for bday
• ISO 8601 time zone for tz
• Telephone numbers requires a numerical form, whilst phone numbers can be presented in alpha-numeric form: e.g. +1-555-FORMATS

hReview

• Uses an ISO 8601 date-time for dtreviewed
• Uses fixed-point integer values from 0-5 for rating (publishers may, for example, display a percentage rating)

hAtom

• Uses ISO 8601 date-time for updated
• Uses ISO 8601 date-time for published

hResume

• Uses ISO 8601 date for individual experience items.
• Uses ISO 8601 date for individual education items.

Geo

• Requires latitude and longitude in decimal form (1.23232;-2.343535), but may be published in degrees: N 37° 24.491, W 122° 08.313
• Locations are most often published just as place names (not abbreviated co-ordinates)

hAudio

• Uses ISO 8601 for track duration, e.g. PT3M23S

Embedding Fixed Data Formats in Microformats

There are currently three supported methods of including these fixed data formats in a microformatted document.

As Visible Page Content

You may use the standard class-design-pattern to mark-up the data visibly in the page.

Ben was born on <span class="bday">1984-02-09</span>.

We're meeting up on Northumberland Avenue (<span class="geo">51.507033,-0.126343</span>).

As An Abbreviation

In some cases, the data formats specified make valid expansions of common human forms, such as dates in in an hCard birthday field:

Ben was born on <abbr class="bday" title="1984-02-09">9th February</abbr>

Note, however, that not all data formats are valid expansions. In HTML, the abbr element is working semantically at a text level, not a data level. Both the abbreviated form (the inner text) and the expanded form (the title) need to be consumable by humans.

This means that in hAudio, using an abbreviation for duration is incorrect:

<abbr class="duration" title="PT3M23S">3 minutes, 23 seconds</abbr>

Whilst the data ‘PT3M23S’ is an expanded form of ‘3 minutes, 23 seconds’, the text is not; ‘PT3M23S’ is nonsense to most human beings. abbr is an element that describes the text, not the data. HTML4 has no way to mark up arbitrary data.

As Supplementary Data using the value-excerption-pattern

The machine data form can be included alongside any human legible text, and hidden using another layer of the browser stack (namely, CSS). This behaviour is documented as the value-excerption-pattern, and derived from the value excerpting behaviour in hCard.

So, for example, when describing a location by name, but still wanting to include geo for the machine-readable location:

<span class="geo">Northumberland Avenue, London <span class="value">51.507033,-0.126343</span></span>

Then, optionally use CSS to hide the data you don't want displayed:

.geo > .value { display: none; }

The same pattern works for the hAudio duration example given above:

<span class="duration">3 minutes and 23 seconds <span class="value">PT3M23S</span></span>

And again, the optional CSS:

.haudio .duration > .value { display: none; }

Of course, this does result in a dependency on CSS to make the data invisible to users, and will result in the machine data being displayed alongside the human form in any user agent without CSS support. That's a compromise that has to be resolved based on the requirements of individual sites.

Proposed Methods

As Invisible Supplementary Data

This section is a proposed extension to value-excerpting, is currently open to active discussion and is not currently supported in parsers. You must not implement this in pages at this time.

Value excerpting is already implemented as a means of extracting data from within a microformat property. Where the element (any element) with class="value" is also empty (containing no inner-text), parsers should instead read the value of the title attribute of that element.

So, the following code will read the inner-text of the element, as per current implementations:

<span class="dtstart">Tomorrow lunchtime <span class="value">2008-05-17T12:00:00+0100</span></span>

The data format, poorly legible to most humans remains visible in the page. To make the machine data invisible at an HTML level, the following can also be parsed:

<span class="dtstart">Tomorrow lunchtime <span class="value" title="2008-05-17T12:00:00+0100"></span></span>

The span with class="value" is empty, therefore the parser must read the value of the title attribute instead. Where the element is not empty, the title attribute is ignored.

Empty elements are invisible within the page, and take up no physical space, so the title is not exposed as a tool-tip. Also, the entire element is ignored by assistive technology such as screen readers, therefore the data is not exposed to any user. This assistive technology claim is based on informed, expert advice, but is awaiting confirmation through testing. That testing is forthcoming.

HTML has no pure way of including machine data inline. The use of value excerpting, which can be applied to any HTML element, is the least obtrusive way to embed data into HTML, without overloading existing element semantics and without browser compatibility issues.

Problems

• Violates the microformats principle of visible data. Numerous previous efforts (e.g. markup in comments etc.) have walked down that path of "dark data" and failed in practice. We must hold ourselves to higher standards than any XML/RDF solution. It's part of what sets microformats apart from so many other failed efforts at data representation on the web. We must not go down the path of dark data. IMHO that principle is inviolable for microformats. Tantek
  • The approach here is that we have exceptional situations where we are requiring data to be duplicated for machines. They are exceptions which have existed in microformats since hCard, and this is a pattern to handle those exceptions and only those exceptions in response to the problems people have publishing them. The specification for this could be written to make it a per-property opt-in device, only for those properties identified above. This is not a ‘generic data embedding’ device and in line with the cited principals, should not be allowed to become one. --BenWard 05:17, 25 Jun 2008 (PDT)
  • An alternative, I suppose, would be to recognise all of the above data format examples as being in violation of the microformats principal, since authors are hiding them in favour of their own content. Every instance of fixed data formats in microformats that force authors to break the invisible data principal would need to be eliminated in favour of accessible, i18n compatible replacements, including those in hCard which are 1:1 mappings from vCard. We _could_ undertake that, but previous discussions (people being advised to misuse ABBR for translation of the vCard telephone types, for example) have already suggested that supporting the visible publishing is too complex. --BenWard 05:17, 25 Jun 2008 (PDT)
• Worsens the DRY violation by separating the human visible version and machine readable version into separate elements. Duplicate data itself is bad, but at least by keeping the duplicates local on the same element (as the existing abbr-pattern does), the risk of drift/divergence is reduced. The greater the distance in content of the duplicates, the greater the risk of drift/divergence, and thus the lower the quality of data. This has been illustrated by the divergence of invisible metadata in the head of a document versus the content in the body, and even more so across documents.
  • The machine-data form is kept as a sibling of the human form, and in distance in code, is not much further away than the data stored on a single elements title attribute. Further, the specification for this could demand the value element be placed as the _first child_ of the parent property, forcing it to be published immediately after the property element. --BenWard 05:17, 25 Jun 2008 (PDT)
• Some parsers (particularly those that run incoming HTML through Tidy to convert it into well-formed XML) may strip empty inline elements. A workaround may be to allow (or even require) hard white space (i.e.  ) within the element with class='value".
  • It is, however, trivial to patch and build Tidy not to do this (keeping empty elements where that element also has a class attribute). Parser writers need to feed back on whether using a custom build is impossible to their solution, but since Tidy can be made to work, the problem can likely be alleviated. Ben Ward has put up an experimental build of Tidy with patched element-dropping behaviour here: tidy-microformats.zip
    • Tidy is not just used in parsers, but also by publishers, as part of CMSes, etc.

Other Proposals

These are proposals and therefore it should be noted that they are not endorsed nor supported. You should not use any of these patterns when publishing a page, but may like to get involved to help develop these ideas.

• Extend the value-excerption-pattern to parse title attributes from empty elements with class value. Empty elements are ignored by browsers in the DOM, so won't render a tool-tip or be exposed to screen readers. Of course, it means putting an empty element into mark-up. Please post responses to this on the value-excerption-pattern-issues page.
• Embed the data into the class attribute, where it will not be rendered or exposed.
• Use a ‘ufusetitle’ class name to particular elements, as a processing instruction to parsers to read the title attribute rather than inner text. However, this adds a new concept of putting ‘parsing instructions’ into the class attribute, which is currently only used to extend the semantics of elements.
• Break with requirements for valid HTML and adopt the RDFa content attribute as a means of embedding data (or another custom attribute). This results in invalid HTML.
  • Depends on what DTD you use. It would not be much work to create a DTD that added a "content" attribute to a few elements. Or simply validate against the RDFa DTD.
• Use the title attribute on any HTML element for data embedding, and prefix the machine data with the string ‘data:’. Users exposed to the data string are given some context. This reduces the impact of the problem at the user side, and assistive technology is less likely to expose the title attribute of generic elements such as span as they are for abbr. This avoids adding empty elements to a page, but continues to expose the machine data to human users through tool-tips.
• Repurpose the input element with a type=hidden attribute. Browsers hide this element from users completely. However, this would stretch the semantics of input; using an input, forms device for output.

Acknowledgements

• James Craig and Bruce Lawson for the suggestion using an empty span as a means of embedding data without it being exposed in assistive technology
• Jeremy Keith for assistance refining the invisible data extension to value-excerpting

Related Pages

• value-excerption-pattern
• class-design-pattern
• abbr-design-pattern
• date-design-pattern
• datetime-design-pattern
• HTML 4.01 definition of <abbr> element