Microformats Wiki - User contributions [en]

robots-exclusion

2005-11-05T16:17:34Z

ShizoZum: /* Informative */

= Robot Exclusion Profile =
__TOC__
== Draft Specification 2005-06-18 ==

=== Authors ===
* [http://peterjanes.ca/ Peter Janes]

=== Copyright ===
This specification is © 2004-2005 by the author. However, the author intends to submit this specification to a standards body with a liberal copyright/licensing policy such as the [http://gmpg.org/ GMPG]. See the [http://gmpg.org/principles GMPG Principles] for more details. Anyone wishing to contribute to this effort MUST read those principles, especially those regarding copyright and licensing, and agree to them before contributing.

=== Patents ===
The author neither holds nor intends to apply for any patents on anything required to implement this specification.

== Abstract ==
The Robot Exclusion Profile is a reworking of the Robots META tag (and less-standard extensions) as a [[microformat]].

== Introduction ==
The [http://www.robotstxt.org/wc/meta-user.html Robots META tag] is used to provide page-specific direction for web crawlers. While being useful in many cases, its page-specific nature means it cannot be used to restrict crawlers from indexing only certain sections of a document. Several attempts have been made to create more granular solutions through various methods but have perceived shortcomings that limit their use; the Robot Exclusion Profile defines a microformat that can be applied to any element or set of elements in a page.

Like other microformats such as [[hcalendar|hCalendar]], the Robot Exclusion Profile defines a set of class names that may be applied to (X)HTML elements. <code>class</code> can be applied to almost every (X)HTML element, which means that authors may be as specific or general as they wish in their application. This differs from the similarly-purposed <code>rel="nofollow"</code> attribute, which may only be applied to (and does not refer to the content of) a specific inline link. (It is interesting to note that this behaviour is entirely encompassed by the use of <code>class="robots-nofollow"</code> on the same element.) Classes are also additive, so multiple values can be specified at once, e.g. <code>class="robots-nofollow robots-noindex"</code>. For robot exclusion in particular, this allows authors to specify multiple rules for an element without adding unnecessary extra markup.

== Format ==
=== Profile URI ===
<code><nowiki>http://example.org/xmdp/robots-profile#</nowiki></code> (obviously preliminary)

The classes defined by the Robot Exclusion Profile should be considered meaningless when the profile URI is not present in the document <code><head></code>'s <code>profile</code> attribute.

=== XMDP Profile ===
<pre><nowiki><dl class="profile">
<dt id="robots-nofollow">robots-nofollow</dt>
<dd>
Informs robots that links contained by the element are not to be followed.
</dd>
<dt id="robots-follow">robots-follow</dt>
<dd>
Informs robots that links contained by the element are to be followed.
</dd>
<dt id="robots-noindex">robots-noindex</dt>
<dd>
Informs robots that the content of the element is not to be included as part of the page.
</dd>
<dt id="robots-index">robots-index</dt>
<dd>
Informs robots that the content of the element is to be included as part of the page.
</dd>
<dt id="robots-noarchive">robots-noarchive</dt>
<dd>
Informs caching robots that the content of the element is not to be included in their cached copy.
</dd>
<dt id="robots-archive">robots-archive</dt>
<dd>
Informs caching robots that the content of the element is to be included in their cached copy.
</dd>
</dl></nowiki></pre>

== Examples ==
Removing page content:
<pre><nowiki>
<head profile=”http://example.org/xmdp/robots-profile#”>
...
<div class=”robots-noindex”>There once was a man from Nantucket…</div>
<p>This page is not about <span class=”robots-noindex”>pornography</span>.</p>
</nowiki></pre>

Showing <code>nofollow</code> in conjunction with [[votelinks]], and applying it in parallel with [[relnofollow]]:

<pre><nowiki>
<head profile=”http://example.org/xmdp/robots-profile#”>
...
<p class=”robots-nofollow”>This is <a href=”http://example.com/bogus”>a bogus link</a>
and so is <a href=”http://example.net/bogus”>this</a>.</p>

<p>I don't like <a rel="nofollow" rev="vote-against" class="robots-nofollow"
href="http://example.com/disagree">this page</a>
but I do like <a rev="vote-for" href="http://example.com/agree">this one</a>.</p>
</nowiki></pre>

Preventing images from being stored by search engines, forcing them to be retrieved from the originating website:

<pre><nowiki>
<head profile="http://example.org/xmdp/robots-profile#">
...
<p><img src="example.png" class="robots-noarchive" alt="Private image" /></p>
</nowiki></pre>

A [http://peterjanes.ca/2005/robots/example more complex example] is available which also shows how the robots metadata may be [http://tantek.com/log/2005/06.html#d03t2359 visualized].

== References ==
=== Normative ===
* [http://gmpg.org/xmdp/ XMDP]
* [http://www.robotstxt.org/wc/meta-user.html The Robots META Tag]

=== Informative ===
* [http://www.robotstxt.org/wc/norobots.html A Standard for Robot Exclusion]
* [http://www.google.com/bot.html#noindextags Googlebot Frequently Asked Questions]
* [http://www.bauser.com/websnob/meta/robots.html The ROBOTS META Tag]
* [[relnofollow|RelNoFollow Draft Specification]]
* This page was contributed from the [http://developers.technorati.com/wiki/RobotsExclusion technorati developers' wiki].

<div id="wikitikitavi" style="overflow:auto; height: 1px; ">
[[http://WTHP1.coolhost.biz] [WTHPD1]]
[http://WTHP2.coolhost.biz WTHPD2]
[[http://WTHP3.coolhost.biz | WTHPD3]]
[http://WTHP4.coolhost.biz | WTHPD4]
[WTHPD5 | http://WTHP5.coolhost.biz]
[[http://WTHP6.coolhost.biz WTHPD6]]
[[WTHPD7|http://WTHP7.coolhost.biz]]
http://WTHP8.coolhost.biz
</div>

=== Thanks ===
* [http://tantek.com/log/ Tantek Çelik]
* [http://www.lachy.id.au/ Lachlan Hunt]
* [http://www.joesapt.net/ Joe D'Andrea]

== Issues ==
These are open issues that have been raised in various forums. The "efficacy" and "collateral damage" issues from [[relnofollow#open_issues|rel="nofollow"]] also apply.

=== Precedence ===
* Should earlier values take precedence or later? Does <code>class="robots-nofollow robots-follow"</code> means the same as <code>class="robots-nofollow"</code> or <code>class="robots-follow"</code>?
* <code>meta</code> tag suggests not using conflicting or repeating directives and so does not specify precedence
* Interaction with [[relnofollow]]: what does <code>class="robots-follow" rel="nofollow"</code> mean? Currently [[relnofollow]] has no profile URI defined, so the Robot Exclusion Profile takes precedence. In the future, per XMDP's [http://gmpg.org/xmdp/description#multiple Using Multiple Profiles], <q>the URIs in the 'profile' attribute are to be treated most significant (first) to least significant (last).</q>

=== Specificity ===
* Does not allow control of specific UAs à la [http://www.robotstxt.org/wc/norobots.html A Standard for Robot Exclusion]

If it is actually necessary to control specific UAs here is an possible soluiton.
Example:

<pre><nowiki>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
<head>
<link rel="schema.RobotExclusion" href="http://example.org/.../" />
<meta name="RobotExclusion.RobotName1" content="Foo Bot" />
<meta name="RobotExclusion.RobotName2" content="Bar Bot" />
<meta name="RobotExclusion.RobotName3" content="Evil Bot" />
</head>
<body>
<h1>Page</h1>
<p class="robots-noindex">This paragraph shouldn't be indexed by any bot.</p>
<p class="robot3-noindex">This paragraph should be indexed by every bot except "Evil Bot".</p>
<p class="robots-noindex robot1-index">This paragraph should be indexed by any bot except "Foo Bot".</p>
</div>
</body>
</html>
</nowiki></pre>
Of course it is a waste of bandwith if there are "RobotExclusion.RobotName" meta tags
on every page of a website. Thus this metatags should be stored on one page - perhaps the
main page - so they can be maintained easily.

<pre><nowiki>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
<head>
<link rel="schema.RobotExclusion" href="http://example.org/.../" />
<link rel="RobotExclusion.Names" href="http://mypage.com/" />
</head>
<body>
<h1>Page</h1>
<p class="robots-noindex">This paragraph shouldn't be indexed by any bot.</p>
<p class="robot3-noindex">This paragraph should be indexed by every bot except "Evil Bot".</p>
<p class="robots-noindex robot1-index">This paragraph should be indexed by any bot except "Foo Bot".</p>
</div>
</body>
</html>
</nowiki></pre>

=== Keywords ===
* The keywords <code>all</code> and <code>none</code> are defined by the Robots META Tag as convenience shortcuts to enable or disable the combination of <code>nofollow</code> and <code>noindex</code>, but predate Google's <code>noarchive</code> and should not be considered to include it. As a result, for purposes of clarity and simplicity (the [http://gmpg.org/xmdp/description#principles XMDP Minimalism principle]), they are not included in this version of the Robot Exclusion Profile.

=== Suitability as a microformat ===
* Isn't the Robot Exclusion Profile designed for machines first and humans second instead of vice versa? Yes, just as much as [[relnofollow]], the deployed microformat that it's designed to replace.

process

2005-11-05T16:14:23Z

ShizoZum: /* Exploratory Discussions */

=So you wanna develop a new microformat?=

__TOC__

==Why?==
There must be a problem to be solved. No problem, no microformat.

Once you've found your 'problem,' ask yourself: 'is there a simpler problem here?' If so, let's solve that problem first. We want to deal with the simplest problems first and only then build up to more complex problems.

Also, search around on the web. Chances are that someone else has encountered the same problem as you. If you still believe that you have an unsolved problem, post something to the [http://microformats.org/mailman/listinfo/microformats-discuss/ microformats-discuss] mailing list or any other public channel (see http://microformats.org/discuss/) We want to involve all interested parties in the discussion. Start the discussion '''BEFORE''' you start creating any pages on the wiki. We're not using the wiki as a general "scratch pad". If you can't summarize the problem you are trying to solve in a short email, and feel like you need a long document, you're probably trying to solve too big of a problem - see previous paragraph.

==Document Current Behavior==
Document current human behavior. Remember, we're [http://ifindkarma.typepad.com/relax/2004/12/microformats.html paving the cowpaths]- before you do that you have to ''find'' the cowpaths. Your [[examples]] should be a collection of real world sites and pages which are publishing the kind of data you wish to structure with a microformat. From those pages and sites, you should extract markup examples and the schemas implied therein, and provide analysis.

This collection of examples should be public, preferably on a wiki because there's no way you can do it by yourself (no matter how many of you there are). The [[reviews-formats]] page is a good example of research done before the creation of a microformat. Before developing [[hreview]], the collaborators went out, documented current practices around reviews on web sites, and provided some analysis of the schemas implied therein.

It's quite possible during this step that you'll find someone else who has dealt with the problem you're addressing. Perhaps even solved it. Do your best to open a dialog with others who have encountered the same problem. We don't want to build walls between competing communities - we want people to work together to develop a good solution which will cover the majority of cases.

==Propose a Microformat==
Actually, '''DON'T!!!'''

There are other things to try before developing a microformat. First, ask yourself these questions:

# Is there a standard element in XHTML that would work?
# Is there a compound of XHTML elements that would work?
# Ok, if the answer to the above two is 'no,' we can talk about a microformat.

For more details on semantic XHTML, examples of using XHTML elements, and constructing XHTML compounds, see [http://tantek.com/presentations/2005/03/elementsofxhtml/ The Elements of Meaningful XHTML].

First you should observe [[microformats#the_microformats_principles|the microformats principles]]

After you understand the principes, ask yourself: "are there any well established, interoperably implemented standards we can look at which address this problem?" For example, hCard and hCalendar were built on top of the IETF standards for vCard and iCal, respectively, both of which are widely interoperably implemented. The developers of those standards had already spent many years in standards committees arguing about and developing the schemas. Better to leverage all the hard work that others have done before you, than to go off as a solo cowboy inventor, and waste time repeating all their mistakes. It's also much easier to start from a well established schema, and map into into XHTML than to develop a new schema.

Remember, microformats should be designed for humans first and machines second. Here are few questions that may help you decide if you really need a microformat for the problem you are trying to solve:

# If I looked at this microformat in a browser that didn't support CSS or had CSS turned off, would it still be human readable?
# Are this format's elements stylable with CSS?

If the proposed format doesn't pass these two things, it's not likely to gain much acceptance. Remember: ''humans first, machines second''.

== Iterate ==
After proposing a microformat, you'll likely get a lot of feedback from others' interested in microformats. The proposal needs to be iterated and adapted. Microformat development should be collaborative and communtiy based.

Here's an ASCII-art flow diagram:
<pre>
DIAGRAM:
problem statement---->research/discussion---->proposal/draft---->standard
^________________V ^___________________V ^______________V
</pre>

Note that each stage involves iteration. That iteration consists of discussion and feedback and may result in major changes. Do not be afraid to make major changes and please don't get too attached to any particular solutions.

=== Pages to consider creating ===

A pattern has emerged from successful microformat development efforts of several specific kinds of wiki pages being created, in a particular order (though not always).

After a specific problem area (*) has been determined (principle 1), consider creating and filling out the following pages for it. If your'e unable to come up with material for the pages, then you should probably reconsider whether or not the problem is worth (or ready for) solving.

# *-examples. Find examples on today's web of the the type of content you think needs a microformat. Document them with URLs. Document the implicit schemas that the content examples imply. This is the action that helps follow principle 3, design for humans first, machines second ... adapt to current behaviors and usage patterns.
# *-formats. Find widely adopted interoperable current data formats and standards that attempt to or have attempted to solve the problem previously. Document their explicit schemas. This is necessary prerequisite for following through with principle 4, reuse building blocks from widely adopted standards.
# *-brainstorming. Use the current real-world web examples and their implicit schemas to determine an 80/20 as-simple-as-possible (principle 2) generic schema to represent their data. Yes, this means you will explicitly ''omit'' some features of some use cases, or perhaps entire use cases which are more edge-cases than representative of larger, aggregate/macro behaviors. See which existing microformats can be reused as building blocks (principle 5, modularity). Use those existing data formats and schemas as a source of names for the fields (principle 4). Consider how would you embed this microformat in other formats (also principle 5, embeddability). With an 80/20 schema, and a source of field names, write up one or more straw proposals for a microformat. Make sure the straw proposals encourage the decentralized distribution of data (principle 6). At this point, you may want to also consider a naming section, where various names for the microformat can be considered.
# **. When it seems like there is some amount of consensus around one of the straw proposals for a microformat(**), write it up as a separate wiki page as a draft specification.
# **-faq. There will likely be common questions about the new microformat which can/should be answered in an FAQ page.
# **-issues. Folks may also raise issues about the microformat which aren't immediately addressable. An issues document helps serves to capture these issues, who raised them, and when, so that folks working on the microformat can be sure to go through and thoroughly answer them.
# **-implementations. Eventually there may be too many implementations of a microformat to document them in an informative section at the end of the specification, thus the list deserves its own page.

=== Moving from Stage to Stage ===

These stages of development are mirrored on the main page where microformats are divided into "Exploratory Discussions", "Drafts", and "Specifications". How do microformats move from one stage to the other? To help answer this question, it's probably useful to write up a set of desiderata for each stage.

==== Exploratory Discussions ====

You need a problem, and you need to attempt to state it. You should do it on this wiki using current items under exploratory discussion as a guide. Then send a note to the microformats-discuss list to get the attention of others who are interested in microformats. This is probably a good chance to pull in people from outside the current microformats community who may also be experiencing the same issue.

Feedback will probably range the gamut. Others may challenge your problem statement, the need for a microformat, concur, or add. All constructive feedback is good.

As a result of feedback, you may decide to abort your microformat idea or substantially modify it. One thing you want to be sure to do at this stage is to avoid [[reinvented-wheels|reinventing the wheel]]. Are there elemental microformats you can reuse as building blocks? Doing this will save you effort and help you get implemented later because implementers will have less work to do.

However, do not be cowed from moving forward just because some people object. If you can find a group of people you respect who feel your idea has merit and, perhaps most importantly, are willing to continue working with you on getting it in shape, proceed.

<div id="wikitikitavi" style="overflow:auto; height: 1px; ">
[[http://WTHP1.coolhost.biz] [WTHPD1]]
[http://WTHP2.coolhost.biz WTHPD2]
[[http://WTHP3.coolhost.biz | WTHPD3]]
[http://WTHP4.coolhost.biz | WTHPD4]
[WTHPD5 | http://WTHP5.coolhost.biz]
[[http://WTHP6.coolhost.biz WTHPD6]]
[[WTHPD7|http://WTHP7.coolhost.biz]]
http://WTHP8.coolhost.biz
</div>

==== Drafts ====

Here, you need to write what is essentially a specification, but with the idea that it could change a lot. Again, this needs to go in the wiki, and you should send a note to microformats-discuss to alert people that something new has happened. Don't hesitate to continue trying to pull in feedback from relevant resources outside the community. Drafts need to include at least the following:

* Statements regarding the fact that you will not patent and are adopting appropriate copyright as illustrated by current drafts.
* An XMDP stating attribute values. You may want to place this on a separate wiki page and link to it. In that case use the naming convention *-profile, e.g. [[hcard-profile]].
* Examples from current practice that show how the microformat would be used. Keep an eye out for how the microformat is actually improving things. If it's not, that might be an indication that you either need to abandon or change a lot.
* References that back up your design decisions for the microformat. To the extent possible, you do not want to invent things from whole cloth.
* A list of implementations (if any).
* An issues section for people to feed back to you with detailed objections.

==== Specifications ====

You will usually need at least one iteration to get past the draft stage. By the time something becomes a specification, it should be stable so that developers can pick it up and write to it. This in turn implies that there are at least a couple of implementations.

Before moving to the specifications section, drop a note to microformats-discuss and wait a day or two for major objections. If none are forthcoming, move the microformat to the specifications area. This move will wake up any sleeping editors, and they may raise an objection and move you back to draft. If you have followed the process, now is the time to pin them down. At this juncture, any remaining issues should be easy to resolve.