Another crucial step to optimising your search engine ranking is ensuring your robots.txt file is present and correct. The robots.txt file is what search engines will look for to make sure they browse the right areas of your site. You may have areas of your site that you don’t want listed or trawled by search engines; If so, the robots.txt file is where you define that, examples of this can include login pages and image folders. You can also define where the sitemap is located, this is recommended; the easier you make it for search engines to trawl your site, the better your site will perform in rankings.
A tool such as Robots Generator, http://www.robotsgenerator.com/ may be useful in creating robots.txt files.