robots-exclusion-issues: Difference between revisions

From Microformats Wiki
Jump to navigation Jump to search
(Undo revision 33770 by PeterJ (Talk) As this is an open issue, I have moved it to the Issues section)
No edit summary
Line 1: Line 1:
<h1> robots exclusion issues </h1>
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
{{TOC-right}}
These are externally raised issues about [[robots exclusion]] with broadly varying degrees of merit. Thus some issues are REJECTED for a number of obvious reasons (but still documented here in case they are re-raised), and others contain longer discussions. Some issues may be ACCEPTED and perhaps cause changes or improved explanations in the spec.


'''IMPORTANT''': Please read the [[robots-exclusion-faq|robots exclusion FAQ]] ''before'' giving any feedback or raising any issues as your feedback/issues may already be resolved/answered.
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST


Submitted issues may (and probably will) be edited and rewritten for better terseness, clarity, calmness, rationality, and as neutral a point of view as possible. Write your issues well. — [http://tantek.com/ Tantek]
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST


For matters relating to the [[meta robots]] specification itself, see [[meta-robots-errata]] and [[meta-robots-suggestions]].
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST


== closed issues ==
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
Resolved issues that have no further actions to take.  These will likely be moved to a separate page like [[robots-exclusion-issues-closed]].


* ...
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST


== resolved issues ==
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
Issues that are resolved but may have outstanding [[to-do]] items. As issues are resolved, they will be moved from the top of the [[robots-exclusion-issues#Issues|Issues list]] to the bottom of this section.
=== 2005 ===
* ''The "efficacy" issues from [[rel-nofollow#open_issues|rel="nofollow"]] also apply.''
** ACCEPTED. The key here would be to get one or more search engines to implement it and get others to do so as well, just as the original [[meta robots]] specification.
* ''The "collateral damage" issues from [[rel-nofollow#open_issues|rel="nofollow"]] also apply.''
** REJECTED EXAMPLE NEEDED.


==== Precedence ====
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
* Should earlier values take precedence or later?  Does <code>class="robots-nofollow robots-follow"</code> means the same as <code>class="robots-nofollow"</code> or <code>class="robots-follow"</code>?
** ACCEPTED. Order {{must not}} matter in usage of the <code>class</code> attribute as it is defined to be a set by HTML4.  The interpretation of conflicting values could be defined conservatively, that is in the presence of both a directive and its "no" variant, the "no" variant supercedes.
* <code>meta</code> tag suggests not using conflicting or repeating directives and so does not specify precedence.  <code>&lt;p class="robots-noindex robots-index"&gt;</code> is an apparent conflict but in this case the more specific should obviously override the general at its point of applicability, no matter what order the directives appear in.
** ACCEPTED FAQ. In this case [[meta robots]] examples should be constructed and tested with current search engines to see how they treat conflicting and/or repeating directives, various combinations, different orderings.
* Interaction with [[rel-nofollow]]: what does <code>class="robots-follow" rel="nofollow"</code> mean? 
** ACCEPTED FAQ. One possible interpretation is that currently [[rel-nofollow]] has no profile URI defined, so the Robot Exclusion Profile takes precedence.  In the future, per XMDP's [http://gmpg.org/xmdp/description#multiple Using Multiple Profiles], <q>the URIs in the 'profile' attribute are to be treated most significant (first) to least significant (last).</q> However XMDP's precedence ordering only applies to the same term being defined twice, not to any sort of semantic conflicts.  Note that [[rel-nofollow]] only states how to treat the link when indexing it, that is, not giving it any weight.  It does not preclude a robot from actually following the link.


==== Phrases ====
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
* Modern search engines normally support <i>phrase</i> queries.  A phrase query only maches documents that contain the words of the query, consecutively and in the same order.  That does beg the question of whether a matched phrase should be allowed to straddle a <code>class="robots-noindex"</code> region. Intuitively this should not be allowed.  The phrase query <code>"word1 word2"</code> should not match a document that contains <code>word1 &lt;b class="robots-noindex&gt;ignore&lt;/b&gt; word2</code>.  This does allow for an interesting tool for webmasters can specify that juxtaposed words not be considered to be phrases -- just specify an empty unindexed region as in <code>word1 &lt;b class="robots-noindex&gt;&lt;/b&gt; word2</code>.
** ACCEPTED FAQ. Yes, any content in a "noindex" element should be treated as if it is not present, and thus would not form part of a phrase.  The second example is less clear however, since many search engines search for phrases withoiut spaces as well, e.g. they provide results for "word1word2" even when the user searches for "word1 word2".


==== Specificity ====
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
* Does not allow control of specific UAs à la [http://www.robotstxt.org/wc/norobots.html A Standard for Robot Exclusion]
** REJECTED OUT OF SCOPE. The [[meta robots]] has no specific control of UAs either, and thus it is out of scope for [[robots-exclusion]].  Brainstorming on a possible extension for specific UAs has been moved to [[robots-exclusion-brainstorming]].


==== Keywords ====
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
* The keywords <code>all</code> and <code>none</code> are defined by the Robots META Tag as convenience shortcuts to enable or disable the combination of <code>nofollow</code> and <code>noindex</code>, but predate Google's <code>noarchive</code> and should not be considered to include it.  As a result, for purposes of clarity and simplicity (the [http://gmpg.org/xmdp/description#principles XMDP Minimalism principle]), they are not included in this version of the Robot Exclusion Profile.
** ACCEPTED FAQ.


== issues ==
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
=== 2006 ===
==== Suitability as a microformat ====
* {{OpenIssue}} Isn't the Robot Exclusion Profile designed for machines first and humans second instead of vice versa?  Yes, just as much as [[rel-nofollow]], the deployed microformat that it's designed to replace.
* I'd like to echo this concern. We need to discuss whether or not this is a suitable microformat. --[[User:RyanKing|RyanKing]] 13:34, 17 Jan 2006 (PST)


=== 2008 ===
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
* {{OpenIssue}} 2008-10-09 raised by [[User: PeterJ|PeterJ]]. Yahoo! has implemented a similar [http://www.ysearchblog.com/2007/05/02/introducing-robots-nocontent-for-page-sections/ robots-nocontent] class.


== template ==
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
Please use this format (copy and paste this to the end of the list to add your issues):
* {{OpenIssue}} YYYY-MM-DD raised by [http://yourhomepage.example.com YOURNAME].
*# ''Issue 1: Here is the first issue I have.''
*# ''Issue 2: Here is the second issue I have.''


== related pages ==
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
* [[robots-exclusion]]
 
* [[robots-exclusion-brainstorming]]
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
 
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
 
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
 
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
 
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
 
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
 
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
 
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
 
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
 
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
 
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
 
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
 
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
 
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
 
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
 
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST
 
I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

Revision as of 04:13, 22 September 2013

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

I MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LISTI MADE THOUSANDS OF ACCOUNTS IN MICROFORMATS WIKI GO TO CHECK OUT THE USER LIST

Retrieved from "http://microformats.org/wiki/index.php?title=robots-exclusion-issues&oldid=63814"