# 20080429 SDM - Updated. # 20070116 SDM - File may be too large for some spiders. # Gutting lots of pages from site so they'll just 404 anyway. # Leaving only * user agent. # 20050826 SDM - Updated to end spidering of non-content dirs. # Repeated, like cnn.com does, for specific robots. User-agent: * Disallow: /errors/ Disallow: /hsc/ Disallow: /kraft/ Disallow: /mint/ Disallow: /nor/ Disallow: /nq/ Disallow: /pure/ Disallow: /r/ Disallow: /static/ Disallow: /trn/ Disallow: /ut/