Main 7 help... Please and Thank you.

Intro 7

purepanic
5 years ago

0

Okay, so I have flown around the threads, and everyone is saying something about robot.txt file. according to **https://en.wikipedia.org/wiki/Robots_exclusion_standard**, A robot.txt file allows you to edit which bots and what they can do on the website in that file. So should i make a web scraper against that page? Or am i headed in the completely wrong direction?
Thank You!

7replies
6voices
310views
dimooz
5 years ago

0

[quote=purepanic]A robot.txt file allows you to edit which bots and what they can do on the website in that file. [/quote]

And sometimes what directories they can access (or not).

purepanic
5 years ago

0

so if i found the robot.txt file and edited it, maybe i could get to the password directory?

dimooz
5 years ago

0

You don’t have many ways to be sure. You’ll have to try.

purepanic
5 years ago

0

Thank You!

Gabriel [airmongeeko]
5 years ago

0

is this function that gives you the text

(function(i, s, o, g, r, a, m) {
i[‘GoogleAnalyticsObject’] = r;
i[r] = i[r] || function() {
(i[r].q = i[r].q || []).push(arguments)
}, i[r].l = 1 * new Date();
a = s.createElement(o),
m = s.getElementsByTagName(o)[0];
a.async = 1;
a.src = g;
m.parentNode.insertBefore(a, m)
})(window, document, ‘script’, ‘//www.google-analytics.com/analytics.js’, ‘ga’);
ga(‘create’, ‘UA-34026704-1’, ‘hackthis.co.uk’);
ga(‘send’, ‘pageview’);

MoSaleh [mosale]
5 years ago

0

@airmongeeko No
This function is not related with the answer

Gabriel [airmongeeko]
5 years ago

0

yup…sorry for the dumb comment that function has to do with visits on the page …

3 replies have been removed
You must be logged in to reply to this discussion. Login
1 of 8

This site only uses cookies that are essential for the functionality of this website. Cookies are not used for tracking or marketing purposes.

By using our site, you acknowledge that you have read and understand our Privacy Policy, and Terms of Service.

Dismiss