robots-exclusion: Difference between revisions

From Microformats Wiki
Jump to navigation Jump to search
(some issues about corner cases)
Line 43: Line 43:
  <dd>
  <dd>
   Informs robots that the content of the element is to be included as part of the page.
   Informs robots that the content of the element is to be included as part of the page.
</dd>
<dt id="robots-noanchortext">robots-noanchortext</dt>
<dd>
  Informs robots that the link target document is not to be indexed under the anchor text.
</dd>
<dt id="robots-anchortext">robots-anchortext</dt>
<dd>
  Informs robots that the link target document is to be indexed under the anchor text.
  </dd>
  </dd>
  <dt id="robots-noarchive">robots-noarchive</dt>
  <dt id="robots-noarchive">robots-noarchive</dt>
Line 83: Line 91:
<p><img src="example.png" class="robots-noarchive" alt="Private image" /></p>
<p><img src="example.png" class="robots-noarchive" alt="Private image" /></p>
</nowiki></pre>
</nowiki></pre>
A consequence of this is that the small summaries that modern search engines display with the result links also exclude the <code>robots-noarchive</code>.  We suggest replacing small excluded segments with an ellipsis [<code>...</code>].  Unarchived segments of a size comparable to the segments the search engine normally uses for summaries can just be omitted.  Probably a display of an entire cached document which has unarchived segments should also include some locution to show the places where text has been elided, no matter what the size.


A [http://peterjanes.ca/2005/robots/example more complex example] is available which also shows how the robots metadata may be [http://tantek.com/log/2005/06.html#d03t2359 visualized].
A [http://peterjanes.ca/2005/robots/example more complex example] is available which also shows how the robots metadata may be [http://tantek.com/log/2005/06.html#d03t2359 visualized].
Line 108: Line 118:
=== Precedence ===
=== Precedence ===
* Should earlier values take precedence or later?  Does <code>class="robots-nofollow robots-follow"</code> means the same as <code>class="robots-nofollow"</code> or <code>class="robots-follow"</code>?
* Should earlier values take precedence or later?  Does <code>class="robots-nofollow robots-follow"</code> means the same as <code>class="robots-nofollow"</code> or <code>class="robots-follow"</code>?
* <code>meta</code> tag suggests not using conflicting or repeating directives and so does not specify precedence
* <code>meta</code> tag suggests not using conflicting or repeating directives and so does not specify precedence.  <code>&lt;p class="robots-noindex robot1-index"&gt;</code> is an apparent conflict but in this case the more specific should obviously override the general at its point of applicability, no matter what order the directives appear in.
* Interaction with [[relnofollow]]: what does <code>class="robots-follow" rel="nofollow"</code> mean?  Currently [[relnofollow]] has no profile URI defined, so the Robot Exclusion Profile takes precedence.  In the future, per XMDP's [http://gmpg.org/xmdp/description#multiple Using Multiple Profiles], <q>the URIs in the 'profile' attribute are to be treated most significant (first) to least significant (last).</q>
* Interaction with [[relnofollow]]: what does <code>class="robots-follow" rel="nofollow"</code> mean?  Currently [[relnofollow]] has no profile URI defined, so the Robot Exclusion Profile takes precedence.  In the future, per XMDP's [http://gmpg.org/xmdp/description#multiple Using Multiple Profiles], <q>the URIs in the 'profile' attribute are to be treated most significant (first) to least significant (last).</q>
=== Phrases ===
Modern search engines normally support <i>phrase</i> queries.  A phrase query only maches documents that contain the words of the query, consecutively and in the same order.  That does beg the question of whether a matched phrase should be allowed to straddle a <code>class="robots-noindex"</code> region.
Intuitively this should not be allowed.  The phrase query <code>"word1 word2"</code> should not match a document that contains <code>word1 &lt;b class="robots-noindex&gt;ignore&lt;/b&gt; word2</code>.  This does allow for an interesting tool for webmasters can specify that juxtaposed words not be considered to be phrases -- just specify an empty unindexed region as in <code>word1 &lt;b class="robots-noindex&gt;&lt;/b&gt; word2</code>.


=== Specificity ===
=== Specificity ===
Line 130: Line 146:
<p class="robots-noindex">This paragraph shouldn't be indexed by any bot.</p>
<p class="robots-noindex">This paragraph shouldn't be indexed by any bot.</p>
<p class="robot3-noindex">This paragraph should be indexed by every bot except "Evil Bot".</p>
<p class="robot3-noindex">This paragraph should be indexed by every bot except "Evil Bot".</p>
<p class="robots-noindex robot1-index">This paragraph should be indexed by any bot except "Foo Bot".</p>
<p class="robots-noindex robot1-index">This paragraph should only be indexed by "Foo Bot".</p>
</div>
</div>
</body>
</body>
Line 150: Line 166:
<p class="robots-noindex">This paragraph shouldn't be indexed by any bot.</p>
<p class="robots-noindex">This paragraph shouldn't be indexed by any bot.</p>
<p class="robot3-noindex">This paragraph should be indexed by every bot except "Evil Bot".</p>
<p class="robot3-noindex">This paragraph should be indexed by every bot except "Evil Bot".</p>
<p class="robots-noindex robot1-index">This paragraph should be indexed by any bot except "Foo Bot".</p>
<p class="robots-noindex robot1-index">This paragraph should only be indexed by "Foo Bot".</p>
</div>
</div>
</body>
</body>

Revision as of 01:36, 2 November 2006

Robot Exclusion Profile

Draft Specification 2005-06-18

Authors

Copyright

This specification is © 2004-2005 by the author. However, the author intends to submit this specification to a standards body with a liberal copyright/licensing policy such as the GMPG. See the GMPG Principles for more details. Anyone wishing to contribute to this effort MUST read those principles, especially those regarding copyright and licensing, and agree to them before contributing.

Patents

The author neither holds nor intends to apply for any patents on anything required to implement this specification.

Abstract

The Robot Exclusion Profile is a reworking of the Robots META tag (and less-standard extensions) as a microformat.

Introduction

The Robots META tag is used to provide page-specific direction for web crawlers. While being useful in many cases, its page-specific nature means it cannot be used to restrict crawlers from indexing only certain sections of a document. Several attempts have been made to create more granular solutions through various methods but have perceived shortcomings that limit their use; the Robot Exclusion Profile defines a microformat that can be applied to any element or set of elements in a page.

Like other microformats such as hCalendar, the Robot Exclusion Profile defines a set of class names that may be applied to (X)HTML elements. class can be applied to almost every (X)HTML element, which means that authors may be as specific or general as they wish in their application. This differs from the similarly-purposed rel="nofollow" attribute, which may only be applied to (and does not refer to the content of) a specific inline link. (It is interesting to note that this behaviour is entirely encompassed by the use of class="robots-nofollow" on the same element.) Classes are also additive, so multiple values can be specified at once, e.g. class="robots-nofollow robots-noindex". For robot exclusion in particular, this allows authors to specify multiple rules for an element without adding unnecessary extra markup.

Format

Profile URI

http://example.org/xmdp/robots-profile# (obviously preliminary)

The classes defined by the Robot Exclusion Profile should be considered meaningless when the profile URI is not present in the document <head>'s profile attribute.

XMDP Profile

<dl class="profile">
 <dt id="robots-nofollow">robots-nofollow</dt>
 <dd>
  Informs robots that links contained by the element are not to be followed.
 </dd>
 <dt id="robots-follow">robots-follow</dt>
 <dd>
  Informs robots that links contained by the element are to be followed.
 </dd>
 <dt id="robots-noindex">robots-noindex</dt>
 <dd>
  Informs robots that the content of the element is not to be included as part of the page.
 </dd>
 <dt id="robots-index">robots-index</dt>
 <dd>
  Informs robots that the content of the element is to be included as part of the page.
 </dd>
 <dt id="robots-noanchortext">robots-noanchortext</dt>
 <dd>
  Informs robots that the link target document is not to be indexed under the anchor text.
 </dd>
 <dt id="robots-anchortext">robots-anchortext</dt>
 <dd>
  Informs robots that the link target document is to be indexed under the anchor text.
 </dd>
 <dt id="robots-noarchive">robots-noarchive</dt>
 <dd>
  Informs caching robots that the content of the element is not to be included in their cached copy.
 </dd>
 <dt id="robots-archive">robots-archive</dt>
 <dd>
  Informs caching robots that the content of the element is to be included in their cached copy.
 </dd>
</dl>

Examples

Removing page content: 

<head profile=”http://example.org/xmdp/robots-profile#”>
...
<div class=”robots-noindex”>There once was a man from Nantucket…</div>
<p>This page is not about <span class=”robots-noindex”>pornography</span>.</p>

Showing nofollow in conjunction with votelinks, and applying it in parallel with relnofollow: 

<head profile=”http://example.org/xmdp/robots-profile#”>
...
<p class=”robots-nofollow”>This is <a href=”http://example.com/bogus”>a bogus link</a>
and so is <a href=”http://example.net/bogus”>this</a>.</p>

<p>I don't like <a rel="nofollow" rev="vote-against" class="robots-nofollow"
                   href="http://example.com/disagree">this page</a>
but I do like <a rev="vote-for" href="http://example.com/agree">this one</a>.</p>

Preventing images from being stored by search engines, forcing them to be retrieved from the originating website: 

<head profile="http://example.org/xmdp/robots-profile#">
...
<p><img src="example.png" class="robots-noarchive" alt="Private image" /></p>

A consequence of this is that the small summaries that modern search engines display with the result links also exclude the robots-noarchive. We suggest replacing small excluded segments with an ellipsis [...]. Unarchived segments of a size comparable to the segments the search engine normally uses for summaries can just be omitted. Probably a display of an entire cached document which has unarchived segments should also include some locution to show the places where text has been elided, no matter what the size.

A more complex example is available which also shows how the robots metadata may be visualized.

References

Normative

Informative

Thanks

Issues

These are open issues that have been raised in various forums. The "efficacy" and "collateral damage" issues from rel="nofollow" also apply.

Precedence

  • Should earlier values take precedence or later? Does class="robots-nofollow robots-follow" means the same as class="robots-nofollow" or class="robots-follow"?
  • meta tag suggests not using conflicting or repeating directives and so does not specify precedence. <p class="robots-noindex robot1-index"> is an apparent conflict but in this case the more specific should obviously override the general at its point of applicability, no matter what order the directives appear in.
  • Interaction with relnofollow: what does class="robots-follow" rel="nofollow" mean? Currently relnofollow has no profile URI defined, so the Robot Exclusion Profile takes precedence. In the future, per XMDP's Using Multiple Profiles, the URIs in the 'profile' attribute are to be treated most significant (first) to least significant (last).

Phrases

Modern search engines normally support phrase queries. A phrase query only maches documents that contain the words of the query, consecutively and in the same order. That does beg the question of whether a matched phrase should be allowed to straddle a class="robots-noindex" region.

Intuitively this should not be allowed. The phrase query "word1 word2" should not match a document that contains word1 <b class="robots-noindex>ignore</b> word2. This does allow for an interesting tool for webmasters can specify that juxtaposed words not be considered to be phrases -- just specify an empty unindexed region as in word1 <b class="robots-noindex></b> word2.

Specificity

If it is actually necessary to control specific UAs here is an possible soluiton. Example: 

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
<head>
<link rel="schema.RobotExclusion" href="http://example.org/.../" />
<meta name="RobotExclusion.RobotName1" content="Foo Bot" />
<meta name="RobotExclusion.RobotName2" content="Bar Bot" />
<meta name="RobotExclusion.RobotName3" content="Evil Bot" />
</head>
<body>
<h1>Page</h1>
<p class="robots-noindex">This paragraph shouldn't be indexed by any bot.</p>
<p class="robot3-noindex">This paragraph should be indexed by every bot except "Evil Bot".</p>
<p class="robots-noindex robot1-index">This paragraph should only be indexed by "Foo Bot".</p>
</div>
</body>
</html>

Of course it is a waste of bandwith if there are "RobotExclusion.RobotName" meta tags on every page of a website. Thus this metatags should be stored on one page - perhaps the main page - so they can be maintained easily. 

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
<head>
<link rel="schema.RobotExclusion" href="http://example.org/.../" />
<link rel="RobotExclusion.Names" href="http://mypage.com/" />
</head>
<body>
<h1>Page</h1>
<p class="robots-noindex">This paragraph shouldn't be indexed by any bot.</p>
<p class="robot3-noindex">This paragraph should be indexed by every bot except "Evil Bot".</p>
<p class="robots-noindex robot1-index">This paragraph should only be indexed by "Foo Bot".</p>
</div>
</body>
</html>

Keywords

  • The keywords all and none are defined by the Robots META Tag as convenience shortcuts to enable or disable the combination of nofollow and noindex, but predate Google's noarchive and should not be considered to include it. As a result, for purposes of clarity and simplicity (the XMDP Minimalism principle), they are not included in this version of the Robot Exclusion Profile.

Suitability as a microformat

  • Isn't the Robot Exclusion Profile designed for machines first and humans second instead of vice versa? Yes, just as much as relnofollow, the deployed microformat that it's designed to replace.
  • I'd like to echo this concern. We need to discuss whether or not this is a suitable microformat. --RyanKing 13:34, 17 Jan 2006 (PST)
Retrieved from "http://microformats.org/wiki/index.php?title=robots-exclusion&oldid=12502"