Well known locations on websites are a bad idea
In other words, the conventions of "/robots.txt" and certainly "/favicon.ico" were mistakes, and contrary to good web/ HTML/ XHTML design.
The W3C TAG has discussed and resolved this issue.
- http://lists.w3.org/Archives/Public/www-tag/2003Feb/0093.html from Tim Berners-Lee
- Stop reading when you get to "Example solution for feasability" and "A new http tag such as "Metadata:" is introduced".