Table of Contents

whitehouse robots

Posted 2003-11-12 @ 16:29:34

Todd Dominey notes that whitehouse.gov has quite a restrictive robots.txt file. Wondering what's left? 


ksmith$ wget -r -l inf -nv http://whitehouse.gov
16:03:15 URL:http://www.whitehouse.gov/ [36587/36587] 
     -> "www.whitehouse.gov/index.html" [1]

FINISHED --16:03:15--
Downloaded: 36,587 bytes in 1 files

The home page. That's it.