csland.com

Independant Computer Consultant

Software Consulting, Development, Strategy, and Programming
Home >  Using <meta> tag for robots and robots.txt

Using <meta> tag for robots and robots.txt

01/15/2010
6287 hits
      Most of robots will check <meta name="robots" content="....."/> and robots.txt file located in the document root to check for instruction to how to index the site. A very simply way is to put these in content:
      
      index, noindex, follow, nofollow, archive, noarchive, snippet, nosnippet, odp, noodp, ydir noydir, imageindex, noimageindex, notranslate, unavailable_after, etc.
      those items can be combined together and separated by comma:
      
      <meta name="robots" content="noindex, follow">
      <meta name="robots" content="index, nofollow">
      <meta name="robots" content="noindex, nofollow">
      
      Very important:
      However, malware robots will ignore your rules, since they scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.

[01/15/2010, 10:14]

References

CSLand.Com © 1996-2009. All Rights Reserved