Your full service technology partner! 
-Collapse +Expand
Search Design Group:

-Collapse +Expand Design Store

Prestwood eMagazine

April Edition
Subscribe now! It's Free!
Enter your email:

   ► KBWebsite Scri...   Print This     
Next Random Article Next Random Tidbit || Change Topic Random From All

Random Tidbit

Part of the mo. lesson in the Jan 16 Issue of Prestwood eMag
Design Tip:
 A flashcard from our Design Flashcards Library
 A tip from our Design Tips Page
Don't Overlook Robots.txt

In the good old days, web spiders would crawl your sites once you registered them with a search engine.  Today, they are a lot more proactive, crawling sites when the domain names are registered.   For this reason, it is not optional during the development phase to add a robots.txt file to all projects that instructs robots not to crawl the site.

It's super easy.  Just create a text file in the website root directory named robots.txt. Put the following text in it.

User-agent: *
Disallow: /

That's it.  Now your temporary website is safe from most webcrawlers.  Note that all subdomains must have one of these in the root path.


Share a thought or comment...
Write a Comment...
Sign in...

If you are a member, Sign In. Or, you can Create a Free account now.

Anonymous Post (text-only, no HTML):

Enter your name and security key.

Your Name:
Security key = P162A1
Enter key:
Tip Contributed By Bryan Valencia:

Bryan Valencia is the author of millions of lines of code and writes for PrestwoodBoards and Visual Studio Journey on-line magazines.

Visit Profile

 KB Article #101795 Counter
Since 1/6/2009
Go ahead!   Use Us! Call: 916-726-5675  Or visit our new sales site:

©1995-2020 Prestwood IT Solutions.   [Security & Privacy]