+ Post New Thread
Results 1 to 4 of 4
Web Development Thread, How Do I Spider Myself in Coding and Web Development; I've made changes to my robots.txt file in the root of my website, but it had me thinking. How can ...
  1. #1
    jmair's Avatar
    Join Date
    Aug 2007
    Thank Post
    Thanked 9 Times in 9 Posts
    Rep Power

    How Do I Spider Myself

    I've made changes to my robots.txt file in the root of my website, but it had me thinking. How can I spider myself and see what results I get? I tried my google foo, however with such a grand word as 'spider' it didnt quite get what I was looking for.

    So how exactly can I Spider myself?

  2. #2

    Join Date
    Jan 2007
    Durham, UK
    Thank Post
    Thanked 17 Times in 12 Posts
    Rep Power
    Try Google Webmaster Tools.

    You can fetch the page as googlebot and view your rankings for different keywords etc...

  3. Thanks to Jackd from:

    jmair (24th March 2010)

  4. #3
    paulst30's Avatar
    Join Date
    Jan 2010
    Thank Post
    Thanked 1 Time in 1 Post
    Rep Power
    There's google webmaster tools, which is useful however i noticed a chrome extension called SEO, and when you go onto your site, you click the SEO button and it gives you stats on how the net see's your site and what search engines have it logged.

    SEO for Chrome

    Obviously you need chrome for it to install. It helped me with my site.

  5. #4

    Join Date
    Nov 2010
    Thank Post
    Thanked 0 Times in 0 Posts
    Rep Power
    there are lots of free tools available where you can check out.. like Search Engine Spider Test Tool ..there are lots more just google search it..

+ Post New Thread

Similar Threads

  1. My First Spider
    By RabbieBurns in forum General Chat
    Replies: 23
    Last Post: 22nd March 2010, 04:19 PM
  2. [Video] Giant Spider Invasion
    By mattx in forum Jokes/Interweb Things
    Replies: 0
    Last Post: 12th December 2009, 04:56 PM
  3. projector spider
    By MrPstv in forum AV and Multimedia Related
    Replies: 6
    Last Post: 11th May 2009, 01:31 PM
  4. New Spider.
    By ICT_GUY in forum General Chat
    Replies: 17
    Last Post: 10th February 2009, 06:39 PM
  5. Who is Yahoo! Slurp Spider & Google Spider?
    By FN-GM in forum General Chat
    Replies: 8
    Last Post: 30th January 2008, 05:45 PM

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts