Microformats Wiki - User contributions [en]

robots-exclusion-issues

2008-10-09T05:38:23Z

PeterJ: Add reference to an implementation of a similar concept in response to a to-do item

<h1> robots exclusion issues </h1>
{{TOC-right}}
These are externally raised issues about [[robots exclusion]] with broadly varying degrees of merit. Thus some issues are REJECTED for a number of obvious reasons (but still documented here in case they are re-raised), and others contain longer discussions. Some issues may be ACCEPTED and perhaps cause changes or improved explanations in the spec.

'''IMPORTANT''': Please read the [[robots-exclusion-faq|robots exclusion FAQ]] ''before'' giving any feedback or raising any issues as your feedback/issues may already be resolved/answered.

Submitted issues may (and probably will) be edited and rewritten for better terseness, clarity, calmness, rationality, and as neutral a point of view as possible. Write your issues well. — [http://tantek.com/ Tantek]

For matters relating to the [[meta robots]] specification itself, see [[meta-robots-errata]] and [[meta-robots-suggestions]].

== closed issues ==
Resolved issues that have no further actions to take. These will likely be moved to a separate page like [[robots-exclusion-issues-closed]].

* ...

== resolved issues ==
Issues that are resolved but may have outstanding [[to-do]] items. As issues are resolved, they will be moved from the top of the [[robots-exclusion-issues#Issues|Issues list]] to the bottom of this section.
=== 2005 ===
* ''The "efficacy" issues from [[rel-nofollow#open_issues|rel="nofollow"]] also apply.''
** ACCEPTED. The key here would be to get one or more search engines to implement it and get others to do so as well, just as the original [[meta robots]] specification.
*** Yahoo! has implemented a similar [http://help.yahoo.com/l/us/yahoo/search/webcrawler/slurp-14.html robots-nocontent] class.
* ''The "collateral damage" issues from [[rel-nofollow#open_issues|rel="nofollow"]] also apply.''
** REJECTED EXAMPLE NEEDED.

==== Precedence ====
* Should earlier values take precedence or later? Does <code>class="robots-nofollow robots-follow"</code> means the same as <code>class="robots-nofollow"</code> or <code>class="robots-follow"</code>?
** ACCEPTED. Order {{must not}} matter in usage of the <code>class</code> attribute as it is defined to be a set by HTML4. The interpretation of conflicting values could be defined conservatively, that is in the presence of both a directive and its "no" variant, the "no" variant supercedes.
* <code>meta</code> tag suggests not using conflicting or repeating directives and so does not specify precedence. <code><p class="robots-noindex robots-index"></code> is an apparent conflict but in this case the more specific should obviously override the general at its point of applicability, no matter what order the directives appear in.
** ACCEPTED FAQ. In this case [[meta robots]] examples should be constructed and tested with current search engines to see how they treat conflicting and/or repeating directives, various combinations, different orderings.
* Interaction with [[rel-nofollow]]: what does <code>class="robots-follow" rel="nofollow"</code> mean?
** ACCEPTED FAQ. One possible interpretation is that currently [[rel-nofollow]] has no profile URI defined, so the Robot Exclusion Profile takes precedence. In the future, per XMDP's [http://gmpg.org/xmdp/description#multiple Using Multiple Profiles], <q>the URIs in the 'profile' attribute are to be treated most significant (first) to least significant (last).</q> However XMDP's precedence ordering only applies to the same term being defined twice, not to any sort of semantic conflicts. Note that [[rel-nofollow]] only states how to treat the link when indexing it, that is, not giving it any weight. It does not preclude a robot from actually following the link.

==== Phrases ====
* Modern search engines normally support <i>phrase</i> queries. A phrase query only maches documents that contain the words of the query, consecutively and in the same order. That does beg the question of whether a matched phrase should be allowed to straddle a <code>class="robots-noindex"</code> region. Intuitively this should not be allowed. The phrase query <code>"word1 word2"</code> should not match a document that contains <code>word1 <b class="robots-noindex>ignore</b> word2</code>. This does allow for an interesting tool for webmasters can specify that juxtaposed words not be considered to be phrases -- just specify an empty unindexed region as in <code>word1 <b class="robots-noindex></b> word2</code>.
** ACCEPTED FAQ. Yes, any content in a "noindex" element should be treated as if it is not present, and thus would not form part of a phrase. The second example is less clear however, since many search engines search for phrases withoiut spaces as well, e.g. they provide results for "word1word2" even when the user searches for "word1 word2".

==== Specificity ====
* Does not allow control of specific UAs à la [http://www.robotstxt.org/wc/norobots.html A Standard for Robot Exclusion]
** REJECTED OUT OF SCOPE. The [[meta robots]] has no specific control of UAs either, and thus it is out of scope for [[robots-exclusion]]. Brainstorming on a possible extension for specific UAs has been moved to [[robots-exclusion-brainstorming]].

==== Keywords ====
* The keywords <code>all</code> and <code>none</code> are defined by the Robots META Tag as convenience shortcuts to enable or disable the combination of <code>nofollow</code> and <code>noindex</code>, but predate Google's <code>noarchive</code> and should not be considered to include it. As a result, for purposes of clarity and simplicity (the [http://gmpg.org/xmdp/description#principles XMDP Minimalism principle]), they are not included in this version of the Robot Exclusion Profile.
** ACCEPTED FAQ.

== issues ==
=== 2006 ===
==== Suitability as a microformat ====
* {{OpenIssue}} Isn't the Robot Exclusion Profile designed for machines first and humans second instead of vice versa? Yes, just as much as [[rel-nofollow]], the deployed microformat that it's designed to replace.
* I'd like to echo this concern. We need to discuss whether or not this is a suitable microformat. --[[User:RyanKing|RyanKing]] 13:34, 17 Jan 2006 (PST)

== template ==
Please use this format (copy and paste this to the end of the list to add your issues):
* {{OpenIssue}} YYYY-MM-DD raised by [http://yourhomepage.example.com YOURNAME].
*# ''Issue 1: Here is the first issue I have.''
*# ''Issue 2: Here is the second issue I have.''

== related pages ==
* [[robots-exclusion]]
* [[robots-exclusion-brainstorming]]

robots-exclusion

2008-03-28T14:07:57Z

PeterJ: /* Copyright */ Link names to user pages

<h1> Robot Exclusion Profile </h1>
{{TOC-right}}
== Draft Specification 2005-06-18 ==

=== Authors ===
* [http://peterjanes.ca/ Peter Janes]

=== Copyright ===
Per the public domain release on the author's and contributors' user pages ([[User:PeterJ|Peter Janes]], [[User:RyanKing|Ryan King]], [[User:Tantek|Tantek Çelik]]) this specification is released into the public domain.

Public Domain Contribution Requirement. Since the author(s) released this work into the public domain, in order to maintain this work's public domain status, all contributors to this page agree to release their contributions to this page to the public domain as well. Contributors may indicate their agreement by adding the public domain release template (http://microformats.org/wiki/Template:public-domain-release) to their user page per the Voluntary Public Domain Declarations instructions (http://microformats.org/wiki/Category:public_domain_license). Unreleased contributions may be reverted/removed.

=== Patents ===
The author neither holds nor intends to apply for any patents on anything required to implement this specification.

== Abstract ==
The Robot Exclusion Profile is a reworking of the [[Robots META]] tag (and less-standard extensions) as a [[microformat]].

== Introduction ==
The [[Robots META]] tag is used to provide page-specific direction for web crawlers. While being useful in many cases, its page-specific nature means it cannot be used to restrict crawlers from indexing only certain sections of a document. Several attempts have been made to create more granular solutions through various methods but have perceived shortcomings that limit their use; the Robot Exclusion Profile defines a microformat that can be applied to any element or set of elements in a page.

Like other microformats such as [[hcalendar|hCalendar]], the Robot Exclusion Profile defines a set of class names that may be applied to (X)HTML elements. <code>class</code> can be applied to almost every (X)HTML element, which means that authors may be as specific or general as they wish in their application. This differs from the similarly-purposed <code>rel="nofollow"</code> attribute, which may only be applied to (and does not refer to the content of) a specific inline link. (It is interesting to note that this behavior is entirely encompassed by the use of <code>class="robots-nofollow"</code> on the same element.) Classes are also additive, so multiple values can be specified at once, e.g. <code>class="robots-nofollow robots-noindex"</code>. For robot exclusion in particular, this allows authors to specify multiple rules for an element without adding unnecessary extra markup.

== Format ==
=== Profile URI ===
<code><nowiki>http://example.org/xmdp/robots-profile#</nowiki></code> (obviously placeholder)

The classes defined by the Robot Exclusion Profile should be considered meaningless when the profile URI is not present in the document <code><head></code>'s <code>profile</code> attribute.

=== XMDP Profile ===
<pre><nowiki><dl class="profile">
<dt id="robots-nofollow">robots-nofollow</dt>
<dd>
Informs robots that links contained by the element are not to be followed.
</dd>
<dt id="robots-follow">robots-follow</dt>
<dd>
Informs robots that links contained by the element are to be followed.
</dd>
<dt id="robots-noindex">robots-noindex</dt>
<dd>
Informs robots that the content of the element is not to be included as part of the page.
</dd>
<dt id="robots-index">robots-index</dt>
<dd>
Informs robots that the content of the element is to be included as part of the page.
</dd>
<dt id="robots-noanchortext">robots-noanchortext</dt>
<dd>
Informs robots that the link target document is not to be indexed under the anchor text.
</dd>
<dt id="robots-anchortext">robots-anchortext</dt>
<dd>
Informs robots that the link target document is to be indexed under the anchor text.
</dd>
<dt id="robots-noarchive">robots-noarchive</dt>
<dd>
Informs caching robots that the content of the element is not to be included in their cached copy.
</dd>
<dt id="robots-archive">robots-archive</dt>
<dd>
Informs caching robots that the content of the element is to be included in their cached copy.
</dd>
</dl></nowiki></pre>

== Examples ==
Removing page content:
<pre><nowiki>
<head profile=”http://example.org/xmdp/robots-profile#”>
...
<div class=”robots-noindex”>There once was a man from Nantucket…</div>
<p>This page is not about <span class=”robots-noindex”>pornography</span>.</p>
</nowiki></pre>

Showing <code>nofollow</code> in conjunction with [[votelinks]], and applying it in parallel with [[rel-nofollow]]:

<pre><nowiki>
<head profile=”http://example.org/xmdp/robots-profile#”>
...
<p class=”robots-nofollow”>This is <a href=”http://example.com/bogus”>a bogus link</a>
and so is <a href=”http://example.net/bogus”>this</a>.</p>

<p>I don't like <a rel="nofollow" rev="vote-against" class="robots-nofollow"
href="http://example.com/disagree">this page</a>
but I do like <a rev="vote-for" href="http://example.com/agree">this one</a>.</p>
</nowiki></pre>

Preventing images from being stored by search engines, forcing them to be retrieved from the originating website:

<pre><nowiki>
<head profile="http://example.org/xmdp/robots-profile#">
...
<p><img src="example.png" class="robots-noarchive" alt="Private image" /></p>
</nowiki></pre>

A consequence of this is that the small summaries that modern search engines display with the result links also exclude the <code>robots-noarchive</code>. We suggest replacing small excluded segments with an ellipsis [<code>...</code>]. Unarchived segments of a size comparable to the segments the search engine normally uses for summaries can just be omitted. Probably a display of an entire cached document which has unarchived segments should also include some locution to show the places where text has been elided, no matter what the size.

A [http://peterjanes.ca/2005/robots/example more complex example] is available which also shows how the robots metadata may be [http://tantek.com/log/2005/06.html#d03t2359 visualized].

== References ==
=== Normative ===
* [http://gmpg.org/xmdp/ XMDP]
* [http://www.robotstxt.org/wc/meta-user.html The Robots META Tag]

=== Informative ===
* [http://www.robotstxt.org/wc/norobots.html A Standard for Robot Exclusion]
* [http://www.google.com/bot.html#noindextags Googlebot Frequently Asked Questions]
* [http://www.bauser.com/websnob/meta/robots.html The ROBOTS META Tag]
* [[relnofollow|RelNoFollow Draft Specification]]
* This page was contributed from the [http://developers.technorati.com/wiki/RobotsExclusion technorati developers' wiki].

=== Thanks ===
* [http://tantek.com/log/ Tantek Çelik]
* [http://www.lachy.id.au/ Lachlan Hunt]
* [http://www.joesapt.net/ Joe D'Andrea]

== related pages ==
* <span id="Issues"> [[robots-exclusion-issues]]</span>
* [[robots-exclusion-brainstorming]]

robots-exclusion

2008-03-28T14:05:37Z

PeterJ: /* Copyright */ Remove copyright statement, since it conflicts with the public domain declaration on the user pages

<h1> Robot Exclusion Profile </h1>
{{TOC-right}}
== Draft Specification 2005-06-18 ==

=== Authors ===
* [http://peterjanes.ca/ Peter Janes]

=== Copyright ===
Per the public domain release on the author's and contributors' user pages (Peter Janes, Ryan King, Tantek Çelik) this specification is released into the public domain.

Public Domain Contribution Requirement. Since the author(s) released this work into the public domain, in order to maintain this work's public domain status, all contributors to this page agree to release their contributions to this page to the public domain as well. Contributors may indicate their agreement by adding the public domain release template (http://microformats.org/wiki/Template:public-domain-release) to their user page per the Voluntary Public Domain Declarations instructions (http://microformats.org/wiki/Category:public_domain_license). Unreleased contributions may be reverted/removed.

=== Patents ===
The author neither holds nor intends to apply for any patents on anything required to implement this specification.

== Abstract ==
The Robot Exclusion Profile is a reworking of the [[Robots META]] tag (and less-standard extensions) as a [[microformat]].

== Introduction ==
The [[Robots META]] tag is used to provide page-specific direction for web crawlers. While being useful in many cases, its page-specific nature means it cannot be used to restrict crawlers from indexing only certain sections of a document. Several attempts have been made to create more granular solutions through various methods but have perceived shortcomings that limit their use; the Robot Exclusion Profile defines a microformat that can be applied to any element or set of elements in a page.

Like other microformats such as [[hcalendar|hCalendar]], the Robot Exclusion Profile defines a set of class names that may be applied to (X)HTML elements. <code>class</code> can be applied to almost every (X)HTML element, which means that authors may be as specific or general as they wish in their application. This differs from the similarly-purposed <code>rel="nofollow"</code> attribute, which may only be applied to (and does not refer to the content of) a specific inline link. (It is interesting to note that this behavior is entirely encompassed by the use of <code>class="robots-nofollow"</code> on the same element.) Classes are also additive, so multiple values can be specified at once, e.g. <code>class="robots-nofollow robots-noindex"</code>. For robot exclusion in particular, this allows authors to specify multiple rules for an element without adding unnecessary extra markup.

== Format ==
=== Profile URI ===
<code><nowiki>http://example.org/xmdp/robots-profile#</nowiki></code> (obviously placeholder)

The classes defined by the Robot Exclusion Profile should be considered meaningless when the profile URI is not present in the document <code><head></code>'s <code>profile</code> attribute.

=== XMDP Profile ===
<pre><nowiki><dl class="profile">
<dt id="robots-nofollow">robots-nofollow</dt>
<dd>
Informs robots that links contained by the element are not to be followed.
</dd>
<dt id="robots-follow">robots-follow</dt>
<dd>
Informs robots that links contained by the element are to be followed.
</dd>
<dt id="robots-noindex">robots-noindex</dt>
<dd>
Informs robots that the content of the element is not to be included as part of the page.
</dd>
<dt id="robots-index">robots-index</dt>
<dd>
Informs robots that the content of the element is to be included as part of the page.
</dd>
<dt id="robots-noanchortext">robots-noanchortext</dt>
<dd>
Informs robots that the link target document is not to be indexed under the anchor text.
</dd>
<dt id="robots-anchortext">robots-anchortext</dt>
<dd>
Informs robots that the link target document is to be indexed under the anchor text.
</dd>
<dt id="robots-noarchive">robots-noarchive</dt>
<dd>
Informs caching robots that the content of the element is not to be included in their cached copy.
</dd>
<dt id="robots-archive">robots-archive</dt>
<dd>
Informs caching robots that the content of the element is to be included in their cached copy.
</dd>
</dl></nowiki></pre>

== Examples ==
Removing page content:
<pre><nowiki>
<head profile=”http://example.org/xmdp/robots-profile#”>
...
<div class=”robots-noindex”>There once was a man from Nantucket…</div>
<p>This page is not about <span class=”robots-noindex”>pornography</span>.</p>
</nowiki></pre>

Showing <code>nofollow</code> in conjunction with [[votelinks]], and applying it in parallel with [[rel-nofollow]]:

<pre><nowiki>
<head profile=”http://example.org/xmdp/robots-profile#”>
...
<p class=”robots-nofollow”>This is <a href=”http://example.com/bogus”>a bogus link</a>
and so is <a href=”http://example.net/bogus”>this</a>.</p>

<p>I don't like <a rel="nofollow" rev="vote-against" class="robots-nofollow"
href="http://example.com/disagree">this page</a>
but I do like <a rev="vote-for" href="http://example.com/agree">this one</a>.</p>
</nowiki></pre>

Preventing images from being stored by search engines, forcing them to be retrieved from the originating website:

<pre><nowiki>
<head profile="http://example.org/xmdp/robots-profile#">
...
<p><img src="example.png" class="robots-noarchive" alt="Private image" /></p>
</nowiki></pre>

A consequence of this is that the small summaries that modern search engines display with the result links also exclude the <code>robots-noarchive</code>. We suggest replacing small excluded segments with an ellipsis [<code>...</code>]. Unarchived segments of a size comparable to the segments the search engine normally uses for summaries can just be omitted. Probably a display of an entire cached document which has unarchived segments should also include some locution to show the places where text has been elided, no matter what the size.

A [http://peterjanes.ca/2005/robots/example more complex example] is available which also shows how the robots metadata may be [http://tantek.com/log/2005/06.html#d03t2359 visualized].

== References ==
=== Normative ===
* [http://gmpg.org/xmdp/ XMDP]
* [http://www.robotstxt.org/wc/meta-user.html The Robots META Tag]

=== Informative ===
* [http://www.robotstxt.org/wc/norobots.html A Standard for Robot Exclusion]
* [http://www.google.com/bot.html#noindextags Googlebot Frequently Asked Questions]
* [http://www.bauser.com/websnob/meta/robots.html The ROBOTS META Tag]
* [[relnofollow|RelNoFollow Draft Specification]]
* This page was contributed from the [http://developers.technorati.com/wiki/RobotsExclusion technorati developers' wiki].

=== Thanks ===
* [http://tantek.com/log/ Tantek Çelik]
* [http://www.lachy.id.au/ Lachlan Hunt]
* [http://www.joesapt.net/ Joe D'Andrea]

== related pages ==
* <span id="Issues"> [[robots-exclusion-issues]]</span>
* [[robots-exclusion-brainstorming]]

User:PeterJ

2008-01-01T01:09:32Z

PeterJ: Correct template

----
{{cc-public-domain-release}}

User:PeterJ

2008-01-01T00:41:35Z

PeterJ: Declare past contributions public domain

----
{{public-domain-release}}

robots-exclusion

2005-06-22T03:00:51Z

PeterJ:

= Robot Exclusion Profile =

== Draft Specification 2005-06-18 ==

=== Authors ===
* [http://peterjanes.ca/ Peter Janes]

=== Copyright ===
This specification is © 2004-2005 by the author. However, the author intends to submit this specification to a standards body with a liberal copyright/licensing policy such as the [http://gmpg.org/ GMPG]. See the [http://gmpg.org/principles GMPG Principles] for more details. Anyone wishing to contribute to this effort MUST read those principles, especially those regarding copyright and licensing, and agree to them before contributing.

=== Patents ===
The author neither holds nor intends to apply for any patents on anything required to implement this specification.

== Abstract ==
The Robot Exclusion Profile is a reworking of the Robots META tag (and less-standard extensions) as a [[microformat]].

== Introduction ==
The [http://www.robotstxt.org/wc/meta-user.html Robots META tag] is used to provide page-specific direction for web crawlers. While being useful in many cases, its page-specific nature means it cannot be used to restrict crawlers from indexing only certain sections of a document. Several attempts have been made to create more granular solutions through various methods but have perceived shortcomings that limit their use; the Robots Exclusion Profile defines a microformat that can be applied to any element or set of elements in a page.

Like other microformats such as [[hcalendar|hCalendar]], the Robot Exclusion Profile defines a set of class names that may be applied to (X)HTML elements. <code>class</code> can be applied to almost every (X)HTML element, which means that authors may be as specific or general as they wish in their application. This differs from the similarly-purposed <code>rel="nofollow"</code> attribute, which may only be applied to (and does not refer to the content of) a specific inline link. (It is interesting to note that this behaviour is entirely encompassed by the use of <code>class="robots-nofollow"</code> on the same element.) Classes are also additive, so multiple values can be specified at once, e.g. <code>class="robots-nofollow robots-noindex"</code>. For robot exclusion in particular, this allows authors to specify multiple rules for an element without adding unnecessary extra markup.

== Format ==
=== Profile URI ===
<code><nowiki>http://example.org/xmdp/robots-profile#</nowiki></code> (obviously preliminary)

The classes defined by the Robot Exclusion Profile should be considered meaningless when the profile URI is not present in the document <code><head></code>'s <code>profile</code> attribute.

=== XMDP Profile ===
<pre><nowiki><dl class="profile">
<dt id="robots-nofollow">robots-nofollow</dt>
<dd>
Informs robots that links contained by the element are not to be followed.
</dd>
<dt id="robots-follow">robots-follow</dt>
<dd>
Informs robots that links contained by the element are to be followed.
</dd>
<dt id="robots-noindex">robots-noindex</dt>
<dd>
Informs robots that the content of the element is not to be included as part of the page.
</dd>
<dt id="robots-index">robots-index</dt>
<dd>
Informs robots that the content of the element is to be included as part of the page.
</dd>
<dt id="robots-noarchive">robots-noarchive</dt>
<dd>
Informs caching robots that the content of the element is not to be included in their cached copy.
</dd>
<dt id="robots-archive">robots-archive</dt>
<dd>
Informs caching robots that the content of the element is to be included in their cached copy.
</dd>
</dl></nowiki></pre>

== Examples ==
Removing page content:
<pre><nowiki>
<head profile=”http://example.org/xmdp/robots-profile#”>
...
<div class=”robots-noindex”>There once was a man from Nantucket…</div>
<p>This page is not about <span class=”robots-noindex”>pornography</span>.</p>
</nowiki></pre>

Showing <code>nofollow</code> in conjunction with [[votelinks]], and applying it in parallel with [[relnofollow]]:

<pre><nowiki>
<head profile=”http://example.org/xmdp/robots-profile#”>
...
<p class=”robots-nofollow”>This is <a href=”http://example.com/bogus”>a bogus link</a>
and so is <a href=”http://example.net/bogus”>this</a>.</p>

<p>I don't like <a rel="nofollow" rev="vote-against" class="robots-nofollow"
href="http://example.com/disagree">this page</a>
but I do like <a rev="vote-for" href="http://example.com/agree">this one</a>.</p>
</nowiki></pre>

Preventing images from being stored by search engines, forcing them to be retrieved from the originating website:

<pre><nowiki>
<head profile="http://example.org/xmdp/robots-profile#">
...
<p><img src="example.png" class="robots-noarchive" alt="Private image" /></p>
</nowiki></pre>

A [http://peterjanes.ca/2005/robots/example more complex example] is available which also shows how the robots metadata may be [http://tantek.com/log/2005/06.html#d03t2359 visualized].

== References ==
=== Normative ===
* [http://gmpg.org/xmdp/ XMDP]
* [http://www.robotstxt.org/wc/meta-user.html The Robots META Tag]

=== Informative ===
* [http://www.robotstxt.org/wc/norobots.html A Standard for Robot Exclusion]
* [http://www.google.com/bot.html#noindextags Googlebot Frequently Asked Questions]
* [http://www.bauser.com/websnob/meta/robots.html The ROBOTS META Tag]
* [[relnofollow|RelNoFollow Draft Specification]]
* This page was contributed from the [http://developers.technorati.com/wiki/RobotsExclusion technorati developers' wiki].

=== Thanks ===
* [http://tantek.com/log/ Tantek Çelik]
* [http://www.lachy.id.au/ Lachlan Hunt]
* [http://www.joesapt.net/ Joe D'Andrea]

== Issues ==
These are open issues that have been raised in various forums. The "efficacy" and "collateral damage" issues from [[relnofollow#open_issues|rel="nofollow"]] also apply.

=== Precedence ===
* Should earlier values take precedence or later? Does <code>class="robots-nofollow robots-follow"</code> means the same as <code>class="robots-nofollow"</code> or <code>class="robots-follow"</code>?
* <code>meta</code> tag suggests not using conflicting or repeating directives and so does not specify precedence
* Interaction with [[relnofollow]]: what does <code>class="robots-follow" rel="nofollow"</code> mean? Currently [relnofollow] has no profile URI defined, so the Robot Exclusion Profile takes precedence. In the future, per XMDP's [http://gmpg.org/xmdp/description#multiple Using Multiple Profiles], <q>the URIs in the 'profile' attribute are to be treated most significant (first) to least significant (last).</q>

=== Specificity ===
* Does not allow control of specific UAs à la [http://www.robotstxt.org/wc/norobots.html A Standard for Robot Exclusion]

=== Keywords ===
* The keywords <code>all</code> and <code>none</code> are defined by the Robots META Tag as convenience shortcuts to enable or disable the combination of <code>nofollow</code> and <code>noindex</code>, but predate Google's <code>noarchive</code> and should not be considered to include it. As a result, for purposes of clarity and simplicity (the [http://gmpg.org/xmdp/description#principles XMDP Minimalism principle]), they are not included in this version of the Robots Exclusion Profile.

robots-exclusion

2005-06-22T02:30:45Z

PeterJ: Add details of inter-microformat precedence, leave intra-format issue open

= Robot Exclusion Profile =

== Draft Specification 2005-06-18 ==

=== Authors ===
* [http://peterjanes.ca/ Peter Janes]

=== Copyright ===
This specification is © 2004-2005 by the author. However, the author intends to submit this specification to a standards body with a liberal copyright/licensing policy such as the [http://gmpg.org/ GMPG]. See the [http://gmpg.org/principles GMPG Principles] for more details. Anyone wishing to contribute to this effort MUST read those principles, especially those regarding copyright and licensing, and agree to them before contributing.

=== Patents ===
The author neither holds nor intends to apply for any patents on anything required to implement this specification.

== Abstract ==
The Robot Exclusion Profile is a reworking of the Robots META tag (and less-standard extensions) as a [[microformat]].

== Introduction ==
The [http://www.robotstxt.org/wc/meta-user.html Robots META tag] is used to provide page-specific direction for web crawlers. While being useful in many cases, its page-specific nature means it cannot be used to restrict crawlers from indexing only certain sections of a document. Several attempts have been made to create more granular solutions through various methods but have perceived shortcomings that limit their use; the Robots Exclusion Profile defines a microformat that can be applied to any element or set of elements in a page.

Like other microformats such as [[hcalendar]], the Robot Exclusion Profile defines a set of class names that may be applied to (X)HTML elements. <code>class</code> can be applied to almost every (X)HTML element, which means that authors may be as specific or general as they wish in their application. This differs from the similarly-purposed <code>rel="nofollow"</code> attribute, which may only be applied to (and does not refer to the content of) a specific inline link. (It is interesting to note that this behaviour is entirely encompassed by the use of <code>class="robots-nofollow"</code> on the same element.) Classes are also additive, so multiple values can be specified at once, e.g. <code>class="robots-nofollow robots-noindex"</code>. For robot exclusion in particular, this allows authors to specify multiple rules for an element without adding unnecessary extra markup.

== Format ==
=== Profile URI ===
<code><nowiki>http://example.org/xmdp/robots-profile#</nowiki></code> (obviously preliminary)

The classes defined by the Robot Exclusion Profile should be considered meaningless when the profile URI is not present in the document <code><head></code>'s <code>profile</code> attribute.

=== XMDP Profile ===
<pre><nowiki><dl class="profile">
<dt id="robots-nofollow">robots-nofollow</dt>
<dd>
Informs robots that links contained by the element are not to be followed.
</dd>
<dt id="robots-follow">robots-follow</dt>
<dd>
Informs robots that links contained by the element are to be followed.
</dd>
<dt id="robots-noindex">robots-noindex</dt>
<dd>
Informs robots that the content of the element is not to be included as part of the page.
</dd>
<dt id="robots-index">robots-index</dt>
<dd>
Informs robots that the content of the element is to be included as part of the page.
</dd>
<dt id="robots-noarchive">robots-noarchive</dt>
<dd>
Informs caching robots that the content of the element is not to be included in their cached copy.
</dd>
<dt id="robots-archive">robots-archive</dt>
<dd>
Informs caching robots that the content of the element is to be included in their cached copy.
</dd>
</dl></nowiki></pre>

== Examples ==
Removing page content:
<pre><nowiki>
<head profile=”http://example.org/xmdp/robots-profile#”>
...
<div class=”robots-noindex”>There once was a man from Nantucket…</div>
<p>This page is not about <span class=”robots-noindex”>pornography</span>.</p>
</nowiki></pre>

Showing <code>nofollow</code> in conjunction with [[votelinks]], and applying it in parallel with [[relnofollow]]:

<pre><nowiki>
<head profile=”http://example.org/xmdp/robots-profile#”>
...
<p class=”robots-nofollow”>This is <a href=”http://example.com/bogus”>a bogus link</a>
and so is <a href=”http://example.net/bogus”>this</a>.</p>

<p>I don't like <a rel="nofollow" rev="vote-against" class="robots-nofollow"
href="http://example.com/disagree">this page</a>
but I do like <a rev="vote-for" href="http://example.com/agree">this one</a>.</p>
</nowiki></pre>

Preventing images from being stored by search engines, forcing them to be retrieved from the originating website:

<pre><nowiki>
<head profile="http://example.org/xmdp/robots-profile#">
...
<p><img src="example.png" class="robots-noarchive" alt="Private image" /></p>
</nowiki></pre>

A [http://peterjanes.ca/2005/robots/example more complex example] is available which also shows how the robots metadata may be [http://tantek.com/log/2005/06.html#d03t2359 visualized].

== References ==
=== Normative ===
* [http://gmpg.org/xmdp/ XMDP]
* [http://www.robotstxt.org/wc/meta-user.html The Robots META Tag]

=== Informative ===
* [http://www.robotstxt.org/wc/norobots.html A Standard for Robot Exclusion]
* [http://www.google.com/bot.html#noindextags Googlebot Frequently Asked Questions]
* [http://www.bauser.com/websnob/meta/robots.html The ROBOTS META Tag]
* [http://developers.technorati.com/wiki/RelNoFollow RelNoFollow Draft Specification]
* This page was contributed from the [http://developers.technorati.com/wiki/RobotsExclusion technorati developers' wiki].

=== Thanks ===
* [http://tantek.com/log/ Tantek Çelik]
* [http://www.lachy.id.au/ Lachlan Hunt]
* [http://www.joesapt.net/ Joe D'Andrea]

== Issues ==
These are open issues that have been raised in various forums. The "efficacy" and "collateral damage" issues from [http://developers.technorati.com/wiki/RelNoFollow#head-0d75f2a2e2ae1ab32cc23dc53d4f46fbc5b3728d rel="nofollow"] also apply.

=== Precedence ===
* Should earlier values take precedence or later? Does <code>class="robots-nofollow robots-follow"</code> means the same as <code>class="robots-nofollow"</code> or <code>class="robots-follow"</code>?
* <code>meta</code> tag suggests not using conflicting or repeating directives and so does not specify precedence
* Interaction with [[relnofollow]]: what does <code>class="robots-follow" rel="nofollow"</code> mean? Currently [relnofollow] has no profile URI defined, so the Robot Exclusion Profile takes precedence. In the future, per XMDP's [http://gmpg.org/xmdp/description#multiple Using Multiple Profiles], <q>the URIs in the 'profile' attribute are to be treated most significant (first) to least significant (last).</q>

=== Specificity ===
* Does not allow control of specific UAs à la [http://www.robotstxt.org/wc/norobots.html A Standard for Robot Exclusion]

=== Keywords ===
* The keywords <code>all</code> and <code>none</code> are defined by the Robots META Tag as convenience shortcuts to enable or disable the combination of <code>nofollow</code> and <code>noindex</code>, but predate Google's <code>noarchive</code> and should not be considered to include it. As a result, for purposes of clarity and simplicity (the [http://gmpg.org/xmdp/description#principles XMDP Minimalism principle]), they are not included in this version of the Robots Exclusion Profile.

robots-exclusion

2005-06-22T01:51:04Z

PeterJ: Correct normative vs. informative references

= Robot Exclusion Profile =

== Draft Specification 2005-06-18 ==

=== Editors ===
* [http://peterjanes.ca/ Peter Janes]
* volunteers welcome

=== Concept ===
* [http://peterjanes.ca/ Peter Janes]

=== Copyright ===
Temporarily © 2004-2005 by the author. However, it is intended to submit this specification to a standards body with a liberal copyright/licensing policy such as the [http://gmpg.org/ GMPG]. See the [http://gmpg.org/principles GMPG Principles] for more details. Anyone wishing to contribute to this effort MUST read those principles, especially those regarding copyright and licensing, and agree to them before contributing.

=== Patents ===
The author neither holds nor intends to apply for any patents on anything required to implement this specification.

== Abstract ==
The Robot Exclusion Profile is a reworking of the Robots META tag (and less-standard extensions) as a [[microformat]].

== Introduction ==
The [http://www.robotstxt.org/wc/meta-user.html Robots META tag] is used to provide page-specific direction for web crawlers. While being useful in many cases, its page-specific nature means it cannot be used to restrict crawlers from indexing only certain sections of a document. Several attempts have been made to create more granular solutions through various methods but have perceived shortcomings that limit their use; the Robots Exclusion Profile defines a microformat that can be applied to any element or set of elements in a page.

Like other microformats such as [[hcalendar]], the Robot Exclusion Profile defines a set of class names that may be applied to (X)HTML elements. <code>class</code> can be applied to almost every (X)HTML element, which means that authors may be as specific or general as they wish in their application. This differs from the similarly-purposed <code>rel="nofollow"</code> attribute, which may only be applied to (and does not refer to the content of) a specific inline link. (It is interesting to note that this behaviour is entirely encompassed by the use of <code>class="robots-nofollow"</code> on the same element.) Classes are also additive, so multiple values can be specified at once, e.g. <code>class="robots-nofollow robots-noindex"</code>. For robot exclusion in particular, this allows authors to specify multiple rules for an element without adding unnecessary extra markup.

== Format ==
=== Profile URL ===
<code><nowiki>http://example.org/xmdp/robots-profile#</nowiki></code> (obviously preliminary)

=== XMDP Profile ===
robots-nofollow:: Informs robots that links contained by the element are not to be followed.
robots-follow:: Informs robots that links contained by the element are to be followed.
robots-noindex:: Informs robots that the content of the element is not to be included as part of the page.
robots-index:: Informs robots that the content of the element is to be included as part of the page.
robots-noarchive:: Informs caching robots that the content of the element is not to be included in their cached copy.
robots-archive:: Informs caching robots that the content of the element is to be included in their cached copy.

== Examples ==
Removing page content:
<pre><nowiki>
<head profile=”http://example.org/xmdp/robots-profile#”>
...
<div class=”robots-noindex”>There once was a man from Nantucket…</div>
<p>This page is not about <span class=”robots-noindex”>pornography</span>.</p>
</nowiki></pre>

Showing <code>nofollow</code> in conjunction with [[votelinks]], and applying it in parallel with [[relnofollow]]:

<pre><nowiki>
<head profile=”http://example.org/xmdp/robots-profile#”>
...
<p class=”robots-nofollow”>This is <a href=”http://example.com/bogus”>a bogus link</a>
and so is <a href=”http://example.net/bogus”>this</a>.</p>

<p>I don't like <a rel="nofollow" rev="vote-against" class="robots-nofollow"
href="http://example.com/disagree">this page</a>
but I do like <a rev="vote-for" href="http://example.com/agree">this one</a>.</p>
</nowiki></pre>

Preventing images from being stored by search engines, forcing them to be retrieved from the originating website:

<pre><nowiki>
<head profile="http://example.org/xmdp/robots-profile#">
...
<p><img src="example.png" class="robots-noarchive" alt="Private image" /></p>
</nowiki></pre>

A [http://peterjanes.ca/2005/robots/example more complex example] is available which also shows how the robots metadata may be [http://tantek.com/log/2005/06.html#d03t2359 visualized].

== References ==
=== Normative ===
* [http://gmpg.org/xmdp/ XMDP]
* [http://www.robotstxt.org/wc/meta-user.html The Robots META Tag]

=== Informative ===
* [http://www.robotstxt.org/wc/norobots.html A Standard for Robot Exclusion]
* [http://www.google.com/bot.html#noindextags Googlebot Frequently Asked Questions]
* [http://www.bauser.com/websnob/meta/robots.html The ROBOTS META Tag]
* [http://developers.technorati.com/wiki/RelNoFollow RelNoFollow Draft Specification]
* This page was contributed from the [http://developers.technorati.com/wiki/RobotsExclusion technorati developers' wiki].

=== Thanks ===
* [http://tantek.com/log/ Tantek Çelik]
* [http://www.lachy.id.au/ Lachlan Hunt]
* [http://www.joesapt.net/ Joe D'Andrea]

== Issues ==
These are open issues that have been raised in various forums. The "efficacy" and "collateral damage" issues from [http://developers.technorati.com/wiki/RelNoFollow#head-0d75f2a2e2ae1ab32cc23dc53d4f46fbc5b3728d rel="nofollow"] also apply.

=== Precedence ===
* Should earlier values take precedence or later? Does <code>class="robots-nofollow robots-follow"</code> means the same as <code>class="robots-nofollow"</code> or <code>class="robots-follow"</code>?
* <code>meta</code> tag suggests not using conflicting or repeating directives and so does not specify precedence
* Interaction with [[relnofollow]]: what does <code>class="robots-follow" rel="nofollow"</code> mean?

=== Specificity ===
* Does not allow blocking of specific UAs à la [http://www.robotstxt.org/wc/norobots.html A Standard for Robot Exclusion]

=== Keywords ===
* The keywords <code>all</code> and <code>none</code> are defined by the Robots META Tag as convenience shortcuts to enable or disable the combination of <code>nofollow</code> and <code>noindex</code>, but predate Google's <code>noarchive</code> and should not be considered to include it. As a result, for purposes of clarity and simplicity, they are not included in this version of the Robots Exclusion Profile.