what is robots.txt?

[rajeshiseo]

please define robot.txt what is it and how it works?

[2WDH.com]

please define robot.txt what is it and how it works?

Hi Rajeshiseo.

robots.txt is a file that can be used by site owner to give instructions to web robots on what/how to crawl the site. It's easy to find a lot of information about this (even if you are searching "robot.txt" (without "s") ).

[zarina]

use google to learn how to create a robots file.

[rajeshiseo]

Hi Rajeshiseo.

robots.txt is a file that can be used by site owner to give instructions to web robots on what/how to crawl the site. It's easy to find a lot of information about this (even if you are searching "robot.txt" (without "s") ).

please sir, give me some detail about robot.txt.it is used in site for allow what crawl in site or what disallow in site?

[redirection]

its a file where you can control which search bot can crawl you blog.
and at the same time you can also configure which part of your site or directory shall be crawled by bots.

the very common config is

User-agent: *
Disallow: /

which means you allow every bots to crawl by indicating the asterisk[*]
forward slash means you allow bots to crawl all from the root

here are some examples when users try to exclude folders not to be crawled

User-agent: *
Disallow: /private/
Disallow: /images-saved/
Disallow: /images-working/

[dailyearner]

it is a file to instruct search engines to crawl your website content

[regmac]

For me, robot.txt is a text file that contains directives for search engines bots that directly affects on how it'll gonna digest your site. For example, if a directive says, disallow blog folder - this would mean that yourdomain.com/blog won't be crawled and indexed. robot.txt file is used to address dupe contents between web pages and directories/folders within a website.

[smith09]

The robots.txt file is a set of instructions for visiting robots (spiders) that index the content of your web site pages.
For those spiders that obey the file, it provides a map of what they can, and cannot index. The file must reside in the root directory of your web. The URL path (web address) of your robots.txt file should look like this...
www. your site name .com/robots.txt

[shopster]

Thanks for this dude.. I also don't know how to use robot.txt.. Thanks a lot for this...

its a file where you can control which search bot can crawl you blog.
and at the same time you can also configure which part of your site or directory shall be crawled by bots.

the very common config is

which means you allow every bots to crawl by indicating the asterisk[*]
forward slash means you allow bots to crawl all from the root

here are some examples when users try to exclude folders not to be crawled

[bishop_ian]

Hello,

Robots.txt is a text file you place on your site to inform search robots which pages you would like them not to visit. Robots.txt is by no means compulsory for search engines but usually search engines follow what they are asked not to do. It is essential to make clear that robots.txt is not a way from stop search engines from crawling your site