Using Robots.txt To Help Google Images Index Your Website.

By Garrett Blanton posted on Thursday, April 8, 2010 - (How To's)

Share |

There’s allot of webmasters out there who utilize the power of the robots.txt file, and use it to block search engine spiders from crawling their cms folders. At JVF we use this method in order to not reveal .js, .css, or any custom coding we use to the search engines. When implementing the Disallow feature within robots.txt we want to warn you that any images within the folder you Disallowed will not be indexed properly within Google Images. To ensure that the photos and images on your website are crawled and indexed properly, be sure to use the Allow rule. You can see an example of this when you take a look at what Google is doing in their robots.txt file. http://www.google.com/robots.txt

They first Disallow anything within their safebrowsing folder, then Allow specific folders which they want crawled and indexed.

Disallow: /safebrowsing
Allow: /safebrowsing/diagnostic
Allow:  /safebrowsing/report_error/
Allow: /safebrowsing/report_phish/

If you were to use this same scenario for your own website using a content management system, it should look something like this:

Disallow: /cms
Allow: /cms/images

Tags:

Robots.txt, Google Images

Comments (6)

By Vilen posted on Saturday, April 10, 2010 @ 4:13 AM

meta name="robots" content="index,follow"
or
User-agent: *
Allow: /

Doesn't it mean that everything are considered for SE crawling? Doesn't it include images either?
Thanks

By houses for sale posted on Tuesday, April 13, 2010 @ 11:57 PM

A robots.txt is a file placed on your server to tell the various search engine spiders not to crawl or index certain sections or pages of your site.It is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web.Among the most important things you can do is check your pages that are in Google's supplemental index. This is where you'll find lots of your low-quality pages, ripe for removal by robots.txt. If the pages don't contain useful information, dump them

By Kristy posted on Wednesday, April 14, 2010 @ 5:30 AM

Hi Vilen. Indeed, but it's generally advised to use Disallow: instead.

The method displayed in this post is to disallow access to a folder *except* image subfolders.

Hope this helps...

By Diagnostic posted on Monday, May 10, 2010 @ 2:49 PM

let' say I have a Folder called FA , and in this forlder there are 100 files, one file I want to allw and rest not, can I type:
Disallow: /FA
Allow : /FA/file.aspx

Thank you

By Rap Music posted on Wednesday, July 20, 2011 @ 10:56 PM

This fix totally saved my website when it comes to organic search results in the Google images section. Everyone who has lots of images on their website should implement this fix immediately.

By Rap Music posted on Wednesday, July 20, 2011 @ 10:57 PM

This fix totally saved my website when it comes to organic search results in the Google images section. Everyone who has lots of images on their website should implement this fix immediately.