o.O

[pne]

(From scribot)

http://www.whitehouse.gov/robots.txt

Um, WTF?

Comments

robots.txt

[pne.livejournal.com]

It also potentially gives away too much information.

I mean, if a person reads robots.txt and sees "Disallow: /company/secret/whitepapers/company-strategy.html", then they know that that URL exists. If you simply "Disallow: /company", then they have much less information.

So it's easiest to put non-robot stuff inside a few directories so you don't have to be as explicit.

Bizarre.

Re: robots.txt

[nik-w.livejournal.com]

Yeah, I imagine they made it with some program or extension to a program (like frontpage or dreamweaver) that just scans your site, lists all the directories with checkboxes next to each, and then makes an entry for each one you check.