What is a robots.txt file?

Intro 7

Diamox
4 years ago | edited 4 years ago

0

I’ve googled it but I don’t fully understand what they are and what they do. Anyone know what they look like and how to access them or is that too much information?
UPDATE: I found a URL link in the robots.txt file but every time I type it in I get ‘error accessing resource’. Am I on the right track or no and if I am how do I get rid of the error?

5replies
3voices
280views
fred [feuerstein]
4 years ago

0

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned. Robots are often used by search engines to categorize websites. Not all robots cooperate with the standard; email harvesters, spambots, malware and robots that scan for security vulnerabilities may even start with the portions of the website where they have been told to stay out. The standard can be used in conjunction with Sitemaps, a robot inclusion standard for websites.

dimooz
4 years ago

0

Usually these files are located at the site root directory. Try this:
https://www.anywebsite.com/robots.txt

example:
https://www.linux.com/robots.txt

Diamox
4 years ago | edited 4 years ago

0

I found the user agents in the robots.txt file and the clear username and password but it doesn’t work. Right now I would like one hint. Is the username and password linked with crawlers?

fred [feuerstein]
4 years ago

0

Tell me via pm what you have please

dimooz
4 years ago

0

Clear username and password work for me. Did you try to copy/paste them or did you type them in?

Discussion thread has been locked. You can no longer add new posts.
1 of 6

This site only uses cookies that are essential for the functionality of this website. Cookies are not used for tracking or marketing purposes.

By using our site, you acknowledge that you have read and understand our Privacy Policy, and Terms of Service.

Dismiss