Results 1 to 6 of 6

Thread: robots and spiders

  1. #1
    Join Date
    Jul 2006
    Location
    Connecticut, USA
    Posts
    2,478

    Default robots and spiders

    can someone please explain in the simplest terms possible, what "robots" and "spiders" are concerning webpages??

    i belong to a sort of networking group for laser light show providers and i keep getting emails from the host saying,

    "a web spider has looked at your site from our site..." something along those lines...

    are these things good? or are they bad?? im assuming its some sort of indexing automated searcher of sorts??

    -Marc
    http://www.laserist.org/images/ildalogos/ILDA-logo_colored-beams_Corporate_150w.jpg

    ILDA- U.S. Laser Regulatory Committee

    Authorized Dealer for:

    • Pangolin Laser Software and Hardware
    • KVANT Laser Modules & Laser Systems
    • X-Laser USA
    • CNI Lasers
    • Cambridge Technology & Eye Magic Professional Scanning Systems

    FDA/CDRH Certified Professional LuminanceRGB Laser Light Show Systems


  2. #2
    Join Date
    Dec 2006
    Location
    Pflugerville, TX, USA
    Posts
    1,977

    Default

    They are used by search engines to index your webpage. That's what makes google work.

  3. #3
    soforene's Avatar
    soforene is offline The Troll formerly known as Herbert Von Poople-Futtocks
    Join Date
    Aug 2007
    Posts
    2,920

    Lightbulb

    Robots (sometimes called spiders) visit web sites and compile an index of the contents for use by search engines.
    The prominence of certain words or repeated use of words means that the robot gives a higher importance to those words and places them higher in its search index.
    Manipulating your site to get it higher in search engines index is an art that some people pay compnies to do.

    You can get more info from http://www.robotstxt.org/faq.html

  4. #4
    Join Date
    Sep 2007
    Location
    Essex, England
    Posts
    795

    Default

    Marc, you should sign up to Googles web master tools...

    First get a google mail account.
    Then log in and go to https://www.google.com/accounts/ManageAccount

    You'll have lots of nice tools at your desposal. Google make it east for first timers to get to grips with their tools.
    You can ad analytics to your pages with a simple line of JavaScript which can produce tones of inetersting data about visitors to your site.

    Graham

  5. #5
    Join Date
    Jul 2006
    Location
    Connecticut, USA
    Posts
    2,478

    Default

    thanks alot everyone!

    graham...

    thanks for that link i am going to do some work with that this evening!

    -Marc
    http://www.laserist.org/images/ildalogos/ILDA-logo_colored-beams_Corporate_150w.jpg

    ILDA- U.S. Laser Regulatory Committee

    Authorized Dealer for:

    • Pangolin Laser Software and Hardware
    • KVANT Laser Modules & Laser Systems
    • X-Laser USA
    • CNI Lasers
    • Cambridge Technology & Eye Magic Professional Scanning Systems

    FDA/CDRH Certified Professional LuminanceRGB Laser Light Show Systems


  6. #6
    Join Date
    Sep 2007
    Location
    Essex, England
    Posts
    795

    Default

    Nice one Mark.

    As a heads up your going to want to use the Analytics mostly. The web tools are interesting but not really impotant untill you start getting lots of links to your site, however it's a good idea to submit a site map in the Webtools section. You can also get some in depth info about your Robots.txt file which is what this thread was about. Google will tell you all the things it can about the robots.txt file as well as any bad links it finds in your site.

    ..I can go on so just get in there and have a play.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •