How Do I Spider Myself

[jmair]

I've made changes to my robots.txt file in the root of my website, but it had me thinking. How can I spider myself and see what results I get? I tried my google foo, however with such a grand word as 'spider' it didnt quite get what I was looking for.

 

So how exactly can I Spider myself?

[Jackd]

Try Google Webmaster Tools.

 

You can fetch the page as googlebot and view your rankings for different keywords etc...

[paulst30]

There's google webmaster tools, which is useful however i noticed a chrome extension called SEO, and when you go onto your site, you click the SEO button and it gives you stats on how the net see's your site and what search engines have it logged.

 

SEO for Chrome

 

Obviously you need chrome for it to install. It helped me with my site.

[mikegayle]

there are lots of free tools available where you can check out.. like Search Engine Spider Test Tool ..there are lots more just google search it..