Birmingham 0121 314 5504
London 0203 397 4288
Cheltenham 0124 232 3088
Manchester 0161 883 1488

Robots.txt Generator Tool

If you wish to inform the robots about the location of your sitemap then simply enter the sitemap URL below.

Example: "http://www.chameleonwebservices.co.uk/sitemap.xml"


Now click "Download" and save the robots.txt file to your local machine and then simply upload to the root of the domain using an FTP client or hosting account file editor.

What are Search Engine Robots

Search Engine Robots are known as Web Crawlers or Spiders, and they are used to automatically travel across the Internet finding information from websites and following links from webpages to find more pages and then following more links.
Search engines such as Google use robots to index web content but robots can be used for negative operations also such as the use by spammers to crawl the Internet looking for email addresses.

How to Disallow or Block using a Robots.txt

In order to disallow or block search engine robots we use a robots.txt file which give us the ability control web robots using the "The Robots Exclusion Protocol".

To control the robots we set specific commands in the robots.txt file. When the robot enters a webpage or follows a link to a webpage the first thing that happens is it will check what is allowed and disallowed.

Example:-

  • Robot enters https://www.chameleonwebservices.co.uk/index.php
  • The robot looks for https://www.chameleonwebservices.co.uk/robots.txt and finds the commands:-

User-agent: *
Disallow: /

The User-agent: * applies to all robots.
The Disallow: / informs the robots to not visit any pages on the site.

How to create a Robots.txt file

Simply use the FREE robots.txt generator above to create your specific instruction set. We have pre-configured data for WordPress and OpenCart, simply click the button relating to the platforms

Chameleon provide this tool for FREE along with many other tools like our FREE SEO Website Checker which show you what a robot sees when it visits your website.

Chameleon Twitter Chameleon Chameleon FaceBook Chameleon Chameleon YouTube Chameleon Chameleon RSS Feed Chameleon Chameleon Blog Chameleon Chameleon Linkedin Chameleon Chameleon Linkedin
Chameleon rated 10 out of 10 based on 36 Client reviews.
Chameleon Support