11

Can user-agents be listed together, followed by their common rules like so in a robots.txt?

User-agent: Googlebot User-agent: ia_archiver #Alexa User-agent: BingPreview #Microsoft User-agent: bingbot #Microsoft User-agent: MSNBot #Microsoft User-agent: Slurp #Yahoo User-agent: Ask Jeeves/Teoma #Ask Disallow: /adm30_buds/
Improve this question
2
  • 2
    Is there a reason why you don't use user-agent: * followed by the disallow? Is the list of "allowed" user-agents longer than the list that you want to disallow? Commented Mar 18, 2014 at 6:24
  • See also on SO: Is it possible to list multiple user-agents in one line? Commented Mar 19, 2014 at 9:28

1 Answer 1

Reset to default
10

Yes. Assuming that your agent names are specified correctly, it looks like this should work. Here is a resource if you want to read more. https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt

Improve this answer
1
  • 2
    The original robots.txt spec agrees: "If more than one User-agent field is present the record describes an identical access policy for more than one robot." Commented Mar 18, 2014 at 15:41

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.