robots-exclusion: Difference between revisions

From Microformats Wiki
Jump to navigation Jump to search
m (Reverted edits by Parpaitas (Talk) to last version by Tantek)
No edit summary
Line 1: Line 1:
<entry-title>Robot Exclusion Profile</entry-title>
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
{{DraftSpecification}}
__TOC__
== Draft Specification 2005-06-18 ==


=== Authors ===
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
* [http://peterjanes.ca/ Peter Janes]


=== Copyright ===
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
Per the public domain release on the author's and contributors' user pages ([[User:PeterJ|Peter Janes]], [[User:RyanKing|Ryan King]], [[User:Tantek|Tantek Çelik]]) this specification is released into the public domain.


Public Domain Contribution Requirement. Since the author(s) released this work into the public domain, in order to maintain this work's public domain status, all contributors to this page agree to release their contributions to this page to the public domain as well. Contributors may indicate their agreement by adding the public domain release template (http://microformats.org/wiki/Template:public-domain-release) to their user page per the Voluntary Public Domain Declarations instructions (http://microformats.org/wiki/Category:public_domain_license). Unreleased contributions may be reverted/removed.
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST


=== Patents ===
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
The author neither holds nor intends to apply for any patents on anything required to implement this specification.


== Abstract ==
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
The Robot Exclusion Profile is a reworking of the [[Robots META]] tag (and less-standard extensions) as a [[microformat]].


== Introduction ==
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
The [[Robots META]] tag is used to provide page-specific direction for web crawlers.  While being useful in many cases, its page-specific nature means it cannot be used to restrict crawlers from indexing only certain sections of a document.  Several attempts have been made to create more granular solutions through various methods but have perceived shortcomings that limit their use; the Robot Exclusion Profile defines a microformat that can be applied to any element or set of elements in a page.


Like other microformats such as [[hcalendar|hCalendar]], the Robot Exclusion Profile defines a set of class names that may be applied to (X)HTML elements.  <code>class</code> can be applied to almost every (X)HTML element, which means that authors may be as specific or general as they wish in their application.  This differs from the similarly-purposed <code>rel="nofollow"</code> attribute, which may only be applied to (and does not refer to the content of) a specific inline link.  (It is interesting to note that this behavior is entirely encompassed by the use of <code>class="robots-nofollow"</code> on the same element.)  Classes are also additive, so multiple values can be specified at once, e.g. <code>class="robots-nofollow robots-noindex"</code>.  For robot exclusion in particular, this allows authors to specify multiple rules for an element without adding unnecessary extra markup.
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST


== Format ==
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
=== Profile URI ===
<code><nowiki>http://example.org/xmdp/robots-profile#</nowiki></code> (obviously placeholder)


The classes defined by the Robot Exclusion Profile should be considered meaningless when the profile URI is not present in the document <code>&lt;head&gt;</code>'s <code>profile</code> attribute.
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST


=== XMDP Profile ===
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
<pre><nowiki><dl class="profile">
<dt id="robots-nofollow">robots-nofollow</dt>
<dd>
  Informs robots that links contained by the element are not to be followed.
</dd>
<dt id="robots-follow">robots-follow</dt>
<dd>
  Informs robots that links contained by the element are to be followed.
</dd>
<dt id="robots-noindex">robots-noindex</dt>
<dd>
  Informs robots that the content of the element is not to be included as part of the page.
</dd>
<dt id="robots-index">robots-index</dt>
<dd>
  Informs robots that the content of the element is to be included as part of the page.
</dd>
<dt id="robots-noanchortext">robots-noanchortext</dt>
<dd>
  Informs robots that the link target document is not to be indexed under the anchor text.
</dd>
<dt id="robots-anchortext">robots-anchortext</dt>
<dd>
  Informs robots that the link target document is to be indexed under the anchor text.
</dd>
<dt id="robots-noarchive">robots-noarchive</dt>
<dd>
  Informs caching robots that the content of the element is not to be included in their cached copy.
</dd>
<dt id="robots-archive">robots-archive</dt>
<dd>
  Informs caching robots that the content of the element is to be included in their cached copy.
</dd>
</dl></nowiki></pre>


== Examples ==
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
Removing page content:
<pre><nowiki>
<head profile=”http://example.org/xmdp/robots-profile#”>
...
<div class=”robots-noindex”>There once was a man from Nantucket…</div>
<p>This page is not about <span class=”robots-noindex”>pornography</span>.</p>
</nowiki></pre>


Showing <code>nofollow</code> in conjunction with [[votelinks]], and applying it in parallel with [[rel-nofollow]]:
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST


<pre><nowiki>
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
<head profile=”http://example.org/xmdp/robots-profile#”>
...
<p class=”robots-nofollow”>This is <a href=”http://example.com/bogus”>a bogus link</a>
and so is <a href=”http://example.net/bogus”>this</a>.</p>


<p>I don't like <a rel="nofollow" rev="vote-against" class="robots-nofollow"
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
                  href="http://example.com/disagree">this page</a>
but I do like <a rev="vote-for" href="http://example.com/agree">this one</a>.</p>
</nowiki></pre>


Preventing images from being stored by search engines, forcing them to be retrieved from the originating website:
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST


<pre><nowiki>
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
<head profile="http://example.org/xmdp/robots-profile#">
...
<p><img src="example.png" class="robots-noarchive" alt="Private image" /></p>
</nowiki></pre>


A consequence of this is that the small summaries that modern search engines display with the result links also exclude the <code>robots-noarchive</code>.  We suggest replacing small excluded segments with an ellipsis [<code>...</code>].  Unarchived segments of a size comparable to the segments the search engine normally uses for summaries can just be omitted.  Probably a display of an entire cached document which has unarchived segments should also include some locution to show the places where text has been elided, no matter what the size.
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST


A [http://peterjanes.ca/2005/robots/example more complex example] is available which also shows how the robots metadata may be [http://tantek.com/log/2005/06.html#d03t2359 visualized].
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST


== References ==
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
=== Normative ===
* [http://gmpg.org/xmdp/ XMDP]
* [http://www.robotstxt.org/wc/meta-user.html The Robots META Tag]


=== Informative ===
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
* [http://www.robotstxt.org/wc/norobots.html A Standard for Robot Exclusion]
* [http://www.google.com/bot.html#noindextags Googlebot Frequently Asked Questions]
* [http://www.bauser.com/websnob/meta/robots.html The ROBOTS META Tag]
* [[relnofollow|RelNoFollow Draft Specification]]
* This page was contributed from the [http://developers.technorati.com/wiki/RobotsExclusion technorati developers' wiki].


=== Thanks ===
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
* [http://tantek.com/log/ Tantek Çelik]
* [http://www.lachy.id.au/ Lachlan Hunt]
* [http://www.joesapt.net/ Joe D'Andrea]


== related pages ==
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
* <span id="Issues"> [[robots-exclusion-issues]]</span>
* [[robots-exclusion-brainstorming]]


[[Category:Draft Specifications]]
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
[[Category:robots-exclusion]]
 
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
 
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
 
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
 
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
 
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
 
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
 
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
 
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

Revision as of 04:13, 22 September 2013

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

Retrieved from "http://microformats.org/wiki/index.php?title=robots-exclusion&oldid=63815"