Printable View
I've got a webpage on a server that I'd rather that google didn't index!
(Don't worry -its just my simple php helpdesk :) )
If there a method of telling search engines to go away and not show it up?
AFAIK its not linked from any other webpage.
regards
Simon
solution found by sticking
"code to avoid google indexer" into google...
(first link)
» 6 ways to stop Google and other search engines from indexing your site | Antezeta Web Marketing
Some good info there.
Chunks
You'll need to use a file called robots.txt in the root of your webserver, you can use it to tell spiders explicitly what they can and can't look at. Dead easy to do, works for all search engines.
Read up about it, and how to do it, @ http://www.robotstxt.org/robotstxt.html
Do the robots.txt but I also add an extra layer myself with this piece of php code for things:
That can just redirect the bots away from your site anyway.Code:$badAgents = array('Mediapartners-Google','msnbot-NewsBlogs/1.1 (+http://search.msn.com/msnbot.htm)','Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp)','Mozilla/5.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)','msnbot/2.0b (+http://search.msn.com/msnbot.htm)','Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)','msnbot/1.1 (+http://search.msn.com/msnbot.htm)', 'Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)');
if(in_array($_SERVER['HTTP_USER_AGENT'],$badAgents)) {
Header("Location: http://BLAH");
exit;
}
Just remember the robots file is a defacto standard that most bots do follow although not all of them do it.
The best way to stop something being spiderd that you don't to be is, is to make sure the bots can't access it, for example applying security to the folder(/s).
Ta for info :)
If there a way of finding out where google might have found the link in the first place
(I probably put it on a pdf/word doc that is in one of my Docman folders but I don't want to have read all of them :( )
regards
Simon
IIRC Google Webmaster tools will show you the incoming links to your site. Have to sign and and verify site ownership (usually by placing a file on the site root or adding meta tags).Quote:
Originally Posted by SimpleSi
Making a robots.txt file seems to have done the trick - my helpdesk has gone dark again :)
regards
Simon
To reitterate a point made earlier - "good" spiders obey robots.txt, but less reputable sites won't. If you don't want the outside world to see parts of the site, make sure they can't see the site. Taking the page off Google isn't going to cut it.
The only problem with using robots.txt is that it can be used maliciously. For example I'd never put the admin directory for my websites listed in robots.txt.
Users could type:Code:www.domainexample.com/robots.txt
*Cough* Microsoft's Bot, although you will know because of other things when it starts to index*cough*Quote:
Originally Posted by SteveBentley
Just look it up and you will see what it means.
..and your suggestion to do this is ? :pQuote:
If you don't want the outside world to see parts of the site, make sure they can't see the site.
regards
Simon
Password protect it or just allow a specific IP range if it's purely for internal use.Quote:
Originally Posted by SimpleSi