I think search engines don’t care much for robot.txt files, they still index it somewhere. By using some of the google advanced operators you can even find webpages with robot.txt.
Reply has been removed
You must be logged in to reply to this discussion.
Login
1 of 4
This site only uses cookies that are essential for the functionality of this website. Cookies are not used for tracking or marketing purposes.