2

I routinely get hits from Microsoft-owned IP addresses in the 13.x.y.z ranges of AS8075 (in these ranges) that present as Bingbot in the User Agent, but these addresses are listed on several block lists, do not pass the Verify Bingbot tool, and do not have hostnames so can't pass the forward-reverse DNS check. Known legit Microsoft bots in other Microsoft IP ranges typically have bi-directional DNS records linking the IP address to a *.search.msn.com hostname and vice versa.

I want to allow legitimate Microsoft bots to crawl my sites, but by all indications these IPs seem to be sketchy.

My question is: Despite indications, are these legitimate Microsoft bot IP addresses, and if these are legitimate Microsoft bots, why wouldn't Microsoft assign hostnames, clean up the malware, and add them to its Verify tool? If these are not actually legitimate Microsoft bots, why wouldn't Microsoft clean up the machines on their network (if they control them), or enforce some rules preventing bad bots on their IPs (if they are rented out to someone else)?

Improve this question

2 Answers 2

Reset to default
2

Looking at Microsoft's public IP address space, it looks to me like Microsoft only owns 13.64.0.0/11, 13.96.0.0/13, and 13.104.0.0/14 - they don't own the entire /8 (13.X.X.X).

The 13.X.X.X block is administered by ARIN, and my guess is that it is divided up among other corporations.

So unless I am mistaken, I don't think Microsoft actually owns the IP space you think they do.

Improve this answer
3
  • I am definitely only talking about IP addresses that trace to Microsoft ownership, in these first four ranges: bgp.he.net/AS8075#_prefixes (slightly broader than you posted). IP lookup of each individual IP address shows each one to be owned by Microsoft. Commented Jan 28, 2019 at 2:28
  • @pseudon That's very interesting. It is possible as you mentioned in your question that these IPs are handed out individually for e.g. Azure applications. However in that case, it would be nearly impossible for Microsoft to enforce any rules on Azure about bot user-agents due to a several technical hurdles, the biggest of which being the TLS encryption of the HTTP request. But I couldn't tell you what would make someone want to spoof these user agents and send their bots to random websites in the first place. Commented Jan 28, 2019 at 2:53
  • Well, there are all kinds of crawlers with all kinds of motivations (and there are a lot of dumb bots too, easily detectable). I assume spoofing a known good bot is to get past overly simplistic filtering rules. To me, a spoofed bot UA is a strong signal that I should block it. You may be right about these IPs being rented out to customers, but it's really odd to me that Miscosoft wouldn't give these addresses unique subdomains or hostnames (like googleusercontent) to distinguish that they are not officially sanctioned Microsoft-controlled IPs. Commented Jan 28, 2019 at 3:12
1

The Verify Bingbot tool isn't reliable, because Bing services don't always use IPs that can be identified.

Site Scan crawler is not yet using Bingbot IP addresses, we plan to switch to Bingbot IP addresses later this year.

That note has been in small text on the Site Scan request page in Bing WM tools for at least 2-3 years. If Microsoft is slacking here for this long, I have no doubt some of the "regular" crawlers aren't identifying properly either.

Improve this answer

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.