Robots.txt Protocol Allows Some Wildcards in Disallow Command

August 10, 2010 – 12:13 am

Robots.txt protocol also supports some wildcard characters in Disallow (or allow) command which can be used for certain form of pattern matching. A asterisk sign (*) is used to specify any number of characters whereas a dollar sign ($) is used to mark the end of any filename. For e.g.

To block folders starting with word private:

        User-agent: Googlebot

        Disallow: /private*/

To block dynamic URLs with question marks in them:

        User-agent: Googlebot

        Disallow: /*?

To block all .pdf files:

        User-agent: *

        Disallow: /*.pdf$

If a ? indicates a session ID, but there are other URLs which end with ? which you  want to index:

        User-agent: *

        Disallow: /*?

        Allow: /*?$