robots.txt content itself is indexed?

Question

The contents of my robots.txt file are actually itself indexed and show up in Google search results. It's only Google and not Yahoo for example.

I really think Google should understand not to index the contents of my robots file as it's only there to tell Google what not to index!

Am I missing something here?

have you ever had a link, on your robot.txt file within the website? — Val, Nov 08 '11 at 10:16

score 2 · Accepted Answer · answered Nov 08 '11 at 10:17

2

That's normal. Mine was indexed a few months ago too. I think it's an issue with the crawler. edit: btw, you can delete the link on the webmaster tools website.

answered Nov 08 '11 at 10:17

sascha

4,671
3
36
54

score 1 · Answer 2 · answered Dec 05 '11 at 08:36

I wouldn't recommend disallowing the robots.txt file from itself. Instead you can use the X-Robots-Tag http header with a value of noindex which will prevent the search engines from Indexing the file.

This will allow them access to your robots.txt but prevent it from showing up in the search results.

score 0 · Answer 3 · answered Nov 28 '11 at 10:47

0

I Think that disallowing your own robots.txt file (if possible) will result in all the other commands written there to be ignored by bots also.

answered Nov 28 '11 at 10:47

Miha

1

If that was the case, putting 'Disallow: *' in the beginning could have the same problem. – Pere Feb 09 '13 at 15:04

score -1 · Answer 4 · answered Nov 08 '11 at 10:19

-1

You can add Disallow: /robots.txt to your robots.txt. I think it should avoid the robots.txt itself from being indexed.

answered Nov 08 '11 at 10:19

acme

14,654
7
75
109

I will try this! Somewhat strange though to disallow something it is already reading. ;-) Thanks! – michael Nov 08 '11 at 11:01

robots.txt content itself is indexed?

4 Answers4