Access Unauthorized Data By Exploiting Robots.txt

In today's blog post, I will be discussing how to Robots.txt can be luring to an attacker and how we can prevent the attacks posed by this.

Robots.txt

Robots.txt is a standard used by websites to inform crawlers and bots about which areas should or should not be scanned. So you can go to any website and visit /robots.txt and it might look something like this:

User-agent: *
Disallow: /*wrappertype=print
Disallow: /*/content/url/
Disallow: /*/content/current/url/
Disallow: /navigation/
Disallow: /error
Disallow: /fragments/
Disallow: /logos/
Disallow: /country-fragment/

Disallow: /admin/password/ 

Disallow: /cgi-bin/
Disallow: /classes/
Disallow: /format/
Disallow: /frames/
Disallow: /db/HELPFILES/
Disallow: /db/MANAGEMENT/
Disallow: /db/MISC/CRICINFO_DATA/
Disallow: /db/SUPPORT/ADVERTS/
Disallow: /db/SUPPORT/AFP/
Disallow: /db/SUPPORT/BSTAR/
Disallow: /db/SUPPORT/DAWN/
Disallow: /db/SUPPORT/DAWSON/
Disallow: /db/SUPPORT/ET/
Disallow: /db/SUPPORT/JAGGED/
Disallow: /db/SUPPORT/SHOP/

So when any crawlers visit the site, they avoid the areas mentioned here.

Such an information is luring to the attacker as it might give them a nice starting point.
As you can see, I see there is a directory called /admin/password. That looks luring. So as an attacker, I will try to go and access the directory directly from my browser and if proper access controls are not in place, I will be able to browse it.
You must be thinking - who will be so stupid to leave it so insecure? Surprisingly, lots of websites do that. And that's where all their passwords are stored :) So the attacker can simply browse to the directory and view or download all the passwords.

Another neat little trick that might help with the directory traversal is as follows:

Try /admin/password. If it doesn't work, try /admin/password/randomNumber&Text/../

The "/../" takes you one directory up. If the website is insecure you might be able to access /admin/password now.

How to Prevent

Having proper access controls for each and every area of your website is a must. For example, a proper admin access for especially sensitive information should be explicitly placed.

Conclusion

So in today's blog post, we saw how robots.txt file can be luring to the attacker and how having proper access controls is a must for a website to be more secure.
For future updates to my weekly blog, please subscribe to my blog via the "Subscribe By Email" feature at the right.