https://microformats.org/wiki/index.php?action=history&feed=atom&title=robots-exclusion-brainstormingrobots-exclusion-brainstorming - Revision history2026-04-26T22:55:36ZRevision history for this page on the wikiMediaWiki 1.38.4https://microformats.org/wiki/index.php?title=robots-exclusion-brainstorming&diff=33751&oldid=prevTantek: drafted, moved a proposal from robots-exclusion-issues to here.2007-11-13T19:33:46Z<p>drafted, moved a proposal from <a href="/wiki/robots-exclusion-issues" title="robots-exclusion-issues">robots-exclusion-issues</a> to here.</p>
<p><b>New page</b></p><div><h1> robots exclusion brainstorming </h1><br />
{{TOC-right}}<br />
This page contains brainstorming, thoughts, and proposals for extending the [[robots-exclusion]] microformat.<br />
<br />
== specific user agents ==<br />
[[robots-exclusion]] lacks the ability to allow control of specific UAs à la [http://www.robotstxt.org/wc/norobots.html A Standard for Robot Exclusion]. While this is out of scope currently since [[meta robots]] has no specific control of UAs either, here are some thoughts that have been proposed:<br />
<br />
If it is actually necessary to control specific UAs here is an possible soluiton.<br />
Example:<br />
<br />
<pre><nowiki><br />
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><br />
<html><br />
<head><br />
<link rel="schema.RobotExclusion" href="http://example.org/.../" /><br />
<meta name="RobotExclusion.RobotName1" content="Foo Bot" /><br />
<meta name="RobotExclusion.RobotName2" content="Bar Bot" /><br />
<meta name="RobotExclusion.RobotName3" content="Evil Bot" /><br />
</head><br />
<body><br />
<h1>Page</h1><br />
<p class="robots-noindex">This paragraph shouldn't be indexed by any bot.</p><br />
<p class="robot3-noindex">This paragraph should be indexed by every bot except "Evil Bot".</p><br />
<p class="robots-noindex robot1-index">This paragraph should only be indexed by "Foo Bot".</p><br />
</div><br />
</body><br />
</html><br />
</nowiki></pre><br />
Of course it is a waste of bandwith if there are "RobotExclusion.RobotName" meta tags<br />
on every page of a website. Thus this metatags should be stored on one page - perhaps the<br />
main page - so they can be maintained easily. <br />
<br />
<pre><nowiki><br />
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><br />
<html><br />
<head><br />
<link rel="schema.RobotExclusion" href="http://example.org/.../" /><br />
<link rel="RobotExclusion.Names" href="http://mypage.com/" /><br />
</head><br />
<body><br />
<h1>Page</h1><br />
<p class="robots-noindex">This paragraph shouldn't be indexed by any bot.</p><br />
<p class="robot3-noindex">This paragraph should be indexed by every bot except "Evil Bot".</p><br />
<p class="robots-noindex robot1-index">This paragraph should only be indexed by "Foo Bot".</p><br />
</div><br />
</body><br />
</html><br />
</nowiki></pre><br />
<br />
=== problems ===<br />
* use of the <code>meta</code> element violates the visibility [[principle]] of microformats.<br />
* encoding content such as "robot1" or "robot3" into class values violates the principle of not putting content into the class attribute.<br />
<br />
=== poll ===<br />
* -1 [[User:Tantek|Tantek]] due to the above problems, and the fact that we have yet to see much adoption of [[robots-exclusion]] in the first place, I think the proposal is both premature and flawed.<br />
<br />
== Habakuk extensions ==<br />
* As I read this, I had the idea to use this microformat to differentiate the real content of a webpage from the rest (navigation, header, footer, ...) - you could do this by marking the "real content" with the tag "index", but thats not really clear. Maybe you could create a new tag to mark the really important things on the page (the "real content") from the rest. --[[User:Habakuk|Habakuk]] 03:42, 14 Jan 2007 (PST)<br />
* And another idea is to mark an area of a page as independent from the rest (p.e. for listings of softwaretools - if i search for an software that can do ''a'' and ''b'' i don't want to get a result that offers me a software that can do ''a'' and another that can do ''b''). --[[User:Habakuk|Habakuk]] 03:42, 14 Jan 2007 (PST)<br />
<br />
== related pages ==<br />
* [[robots-exclusion]]<br />
* [[robots-exclusion-issues]]</div>Tantek