What is a robots.txt file?
A "robot", "bot", or "spider" are computer programs that crawl the internet for a variety of purposes. These bots can be useful (Googlebot crawling your website to add to the Google search engine), or they can be undesirable (bots crawling your website looking for forms and email addresses to send SPAM to for example).
A robots.txt file is a small text file which contains information about which areas of a website the website's owner would like the bots would like to crawl and which areas should be off limits. Please note that the areas you add as off-limits in the robots.txt file are only optional for bots to follow - nefarious bots may ignore the file completely and continue to crawl pages of your website you disallowed in the robots.txt file.
For more detailed information about bots, what a robots.txt file is, and the information to add to it, please see this website: http://www.robotstxt.org
How to upload a custom robots.txt file
Uploading a robots.tx file works exactly the same as uploading any file to the root of your website. Here are the steps again for convenience:
- Login to the Admin area of your site
- Navigate to Developers-> (FTP) File Manager
- Click on “Upload”
- Click on “Choose File” or “Browse” (could be different depending on what browser you are using)
- Choose the robots.txt file on your hard drive
- Click “Open” and the file will now be uploaded automatically
- Once the upload is complete, click on the "Back to..." link on the bottom of the screen, or simply close the browser tab you were using to upload the file in order to return to the File Manager
- The file will now be accessible at http://www.YourDomain.com/robots.txt