Googlebot ignoring robots.txt
I have the standard robots.txt file on my server, Googlebot loads it ok (I see the hit), but then follows a load of hits like:
66.249.66.194 - - 27/Jan/2005:22:33:28 +0000 "GET /tiki-pagehistory.php?page=The+Hermit&diff2= HTTP/1.1" 200 10441 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
tiki-pagehistory.php is in the robots.txt as disallow, so why is Googlebot still crawling it?
Anyone else noticed this?
- Gray