# but allow only important bots User-agent: Googlebot Disallow: User-agent: Googlebot-Image Disallow: User-agent: googlebot-mobile Disallow: User-agent: Mediapartners-Google Disallow: User-agent: bingbot Disallow: User-agent: msnbot Disallow: User-agent: msnbot-media Disallow: User-agent: Slurp Disallow: User-agent: Yahoo-Blogs Disallow: User-agent: Yahoo-MMCrawler Disallow: User-agent: Teoma Disallow: User-agent: Robozilla Disallow: User-agent: Yahoo-VerticalCrawler Disallow: User-agent: psbot Disallow: User-agent: online-webceo-bot Disallow: User-agent: Twitterbot Disallow: User-agent: Grapeshot Disallow: User-agent: Grapeshot Allow: * User-agent: Twitterbot Allow: *