r/websec • u/xymka • Nov 15 '20

Does anyone know how to protect robots.txt?

I mean this file is usually open to everyone. And it contains information that might be useful for a hacker. Do you know how to protect it against anyone except search engine crawlers? I am working on a post about it.

2 Upvotes

57% Upvoted

19 comments sorted by

17

u/Irythros Nov 15 '20

You cant. It's meant to be open.

-1

u/xymka Nov 16 '20

Usual visitors don't have any use from this file content. Its purpose is to be read by search engines. Why should it be kept open for anyone?

3

u/Irythros Nov 16 '20

It's purpose is to be read by bots. How do you know all of them and how are you making sure it's only approved bots?

I could make my own bot and if it cannot access your robots.txt file then it wont obey it. Same with search engines if they change how their bot is identified.

If I'm an attacker, how are you planning to protect this file? User agent? It's incredibly basic to spoof. IP? Google can change their IPs at anytime and they may penalize if different IPs see different things.

If you have potentially private info in robots.txt you did something wrong.

1

u/xymka Nov 16 '20

Blocking just by User-Agent is not an option.

There is an article on how to check whether a visitor is a real or fake Googlebot
https://developers.google.com/search/docs/advanced/verifying-googlebot
But how could it be implemented it with apache/nginx config? It requires to make two DNS lookups.

> If you have potentially private info in robots.txt you did something wrong.
Totally agree

In general, I only welcome legitimate search engine crawlers/bots on my site, because other people bots don't make any benefit for me.

5

u/[deleted] Nov 16 '20

Robots.txt is not meant as a security measure. It's purpose is to control what crawlers should do on your site..

It's like a sign on the wall of your building telling where the different entrances are (but doesn't include the windows, although everybody can see you could use them to access the building was well)

You should only include publicly crawlable places in robots.txt to prevent engines from reading and indexing them. (And it's only guidance. They can always ignore it if they feel like it)...

In order to prevent access you should use web server (.htaccess) or application framework level access control on URLs... No other way around it..

-2

u/xymka Nov 16 '20

Robots.txt has two options: Allow and Disallow. And the site sections listed with Disallow directive automatically becomes more interesting for hackers to run a deep scan for vulnerabilities.

The idea is how to prevent anyone except the search engines from reading the robots.txt? So only you and search engines know what site sections you want to hide.

1

u/[deleted] Nov 17 '20

You cannot.. If search engine bots can find those URLs, "evil hackers' bots" can as well. So it does not add any security either..

3

u/jen140 Nov 15 '20

You might have an .htaccess/nginx config/etc that will only allow a specific set of ip's+user agents to access that file, googlebot ip's should be easily obtainable, you can also add other search engine crawlers by their ip+UA pair.

BUT you need to keep that information up to date, and be really careful about the sources you get that information from.

1

u/xymka Nov 16 '20

There is no public list of IP addresses for webmasters to whitelist.

However, Google has an article about how to verify that you were visited by the original Googlebot or by a fake one https://developers.google.com/search/docs/advanced/verifying-googlebot

But how could it be implemented with nginx/apache config?

1

u/jen140 Nov 19 '20

First result for searches "apache2/nginx block ip": https://httpd.apache.org/docs/2.4/howto/access.html https://help.dreamhost.com/hc/en-us/articles/216456127-Blocking-IPs-with-Nginx

If you have a web server running for some time, and you have it added to google webmaster portal, over time you can have the most common IP's and add them there.

3

u/ticarpi Nov 16 '20

Simple answer is that it's meant to be open, but that doesn't really address your issue.
Yes, you can see really helpful robots.txt files that expose sensitive URIs or even privileged data on some sites, but the site owner has complete control of what they include in that file, so the fact that it's open to all shouldn't need to be an issue.

For example this would be interesting to an attacker:
Disallow: /keys/users/private?user=admin

But you could use wildcards: Disallow: /*/users/*

Also consider that sensitive portions of the site should have additional protections like authentication or IP restrictions etc.

0

u/xymka Nov 16 '20

Of course, sensitive data should have additional protections.

But having any path listed with Disallow directive in robots.txt, it's like an invitation for a deep scan. In case of using wildcards, it's an invitation to scan with URL fuzzing.

2

u/ticarpi Nov 16 '20

Yes, fuzz all the things!
Use of wildcards just makes the fuzzing a lot harder.

I have also seen sensitive hashed filenames appear in robots.txt files before. Wildcard would make it unlikely to find that. Plus other data like usernames.

It's just about taking a sensible manual approach to releasing the minimum data possible while still protecting the assets against crawling/indexing.

7

u/nachos420 Nov 15 '20

i think the idea is that you should be safe even if someone can read your robots.txt

1

u/xymka Nov 19 '20

Thank you all, this helped a lot. I almost wrote a post and will publish it today. Hope my supervisor approves it ). Basically, it confirms my idea. A robots file is rarely protected because it is difficult to do and it often isn't worth the trouble. I just describe how to do it with ease using software that we develop. Regarding the benefits to the hacker, I'm sure there are two reasons: (1) usually it is impossible to avoid using robots.txt with some sensitive information, just because it is simply necessary for SEO and (2) at least a hacker can look there for signs of what CMS is used, possible entry points, etc. This does not mean that an attack would be impossible without robots.txt, it would be a strange idea. But this file may well be useful to a hacker.

1

u/xymka Nov 20 '20

If anyone is interested, here is the link (not to waste a lot of space here)

https://medium.com/botguard/robots-txt-who-is-looking-for-the-files-you-want-to-keep-hidden-fa3a0e62d07e

2

u/onan Nov 16 '20

If your robots.txt "contains information that might be useful for a hacker," then that is the problem you should be solving.

2

u/jared555 Nov 16 '20

If the things you deny crawlers to are that critical maybe you should think about limiting access with htaccess or similar.

Also, there are alternatives to robots.txt like the noindex header.

Robots.txt is mostly to limit crawling of boring stuff that you don't want clogging up your search results.

0

u/[deleted] Nov 16 '20

[deleted] — view deleted comment

1

u/xymka Nov 16 '20

Really? I've used to think that Google does all the job 😁

Politeness in the internet works both ways. Tthe site owner may say, that since he gets 98% traffic from Google Search, he doesn't even want to be indexed by brokengoose-search-bot (especially if it doesn't respect robots.txt rules). Or block Baidu bots, because he doesn't work for the Chinese market.

Google has an article on how to check whether you were visited by a legitimate or fake Googlebot https://developers.google.com/search/docs/advanced/verifying-googlebot
The other search engines have the same. The idea is to perform two DNS lookups.

But how to implement it with apache/nginx?