making a webpage invisible to Google

[SimpleSi]

I've got a webpage on a server that I'd rather that google didn't index!

 

(Don't worry -its just my simple php helpdesk )

 

If there a method of telling search engines to go away and not show it up?

 

AFAIK its not linked from any other webpage.

 

regards

 

Simon

[plexer]

Removing my own content from Google - Webmaster Tools Help

[Chunky]

solution found by sticking

"code to avoid google indexer" into google...

 

(first link)

 

» 6 ways to stop Google and other search engines from indexing your site | Antezeta Web Marketing

 

Some good info there.

 

Chunks

[sonofsanta]

You'll need to use a file called robots.txt in the root of your webserver, you can use it to tell spiders explicitly what they can and can't look at. Dead easy to do, works for all search engines.

 

Read up about it, and how to do it, @ http://www.robotstxt.org/robotstxt.html

[ZeroHour]

Do the robots.txt but I also add an extra layer myself with this piece of php code for things:

$badAgents = array('Mediapartners-Google','msnbot-NewsBlogs/1.1 (+http://search.msn.com/msnbot.htm)','Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp)','Mozilla/5.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)','msnbot/2.0b (+http://search.msn.com/msnbot.htm)','Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)','msnbot/1.1 (+http://search.msn.com/msnbot.htm)', 'Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)');
if(in_array($_SERVER['HTTP_USER_AGENT'],$badAgents)) {    
Header("Location: http://BLAH");
exit;
}

 

That can just redirect the bots away from your site anyway.

[p858snake]

Just remember the robots file is a defacto standard that most bots do follow although not all of them do it.

 

The best way to stop something being spiderd that you don't to be is, is to make sure the bots can't access it, for example applying security to the folder(/s).

[SimpleSi]

Ta for info

 

If there a way of finding out where google might have found the link in the first place

 

(I probably put it on a pdf/word doc that is in one of my Docman folders but I don't want to have read all of them )

 

regards

 

Simon

[pwds]

If there a way of finding out where google might have found the link in the first place

 

IIRC Google Webmaster tools will show you the incoming links to your site. Have to sign and and verify site ownership (usually by placing a file on the site root or adding meta tags).

[SimpleSi]

Making a robots.txt file seems to have done the trick - my helpdesk has gone dark again

 

regards

 

Simon

[SteveBentley]

To reitterate a point made earlier - "good" spiders obey robots.txt, but less reputable sites won't. If you don't want the outside world to see parts of the site, make sure they can't see the site. Taking the page off Google isn't going to cut it.

[Michael]

The only problem with using robots.txt is that it can be used maliciously. For example I'd never put the admin directory for my websites listed in robots.txt.

 

Users could type:

www.domainexample.com/robots.txt

[p858snake]

To reitterate a point made earlier - "good" spiders obey robots.txt, but less reputable sites won't. If you don't want the outside world to see parts of the *site, make sure they can't see the site. Taking the page off Google isn't going to cut it.

*Cough* Microsoft's Bot, although you will know because of other things when it starts to index*cough*

 

Just look it up and you will see what it means.

[SimpleSi]

If you don't want the outside world to see parts of the site, make sure they can't see the site.

..and your suggestion to do this is ?

 

regards

 

Simon

[SteveBentley]

..and your suggestion to do this is ?

 

Password protect it or just allow a specific IP range if it's purely for internal use.

[plexer]

http://google.co.uk/robots.txt

 

Ben

[pete]

..and your suggestion to do this is ?

 

regards

 

Simon

 

SSL and a .htaccess?