Bing Opens Web IQ to All Azure Users; 107 of 128 Sites That Block AI Bots Admit Bingbot
Microsoft made Web IQ, its Bing-powered grounding service for AI agents, available to all Azure customers on 17 November, fed by Bingbot's index. Our crawl of 508 top-site robots.txt files found 107 of the 128 sites that block AI crawlers still let Bingbot in, and one homepage using Bing's grounding opt-out.

Ras Al Khaimah, United Arab Emirates, .
Microsoft made Web IQ, its Bing-powered search and grounding service for AI agents, available to every Azure customer on 17 November 2026. In the announcement on the Bing blog, Jordi Ribas, Microsoft's president of Search and AI, said the service draws on Microsoft's own index, built by Bingbot, the same crawler that serves Bing Search and Copilot. That makes Bingbot access the gate to a growing number of third-party AI agents.
To see how many sites have closed that gate, we pulled the robots.txt files of the 1,000 most popular domains on the Tranco list on 18 November. Of 508 unique files, 128 block at least one of four major AI crawlers (GPTBot, ClaudeBot, PerplexityBot or OAI-SearchBot) from the whole site. Only 21 of those 128 also block Bingbot, so 107 leave it free to crawl, among them The New York Times, CNN, the BBC, Reuters and Bloomberg. Just one homepage we checked uses Bing's new tag for opting out of AI grounding.
What changed
- General availability. Web IQ, first announced in June, is now open to all Azure customers at what Microsoft calls "new, lower pricing", billed through Azure. The blog does not publish the new prices.
- Auto becomes the default. One Auto call can search web, news, images, video, places and weather sources and return cited passages rather than whole pages. An Auto Advanced mode runs extra retrieval passes when the first results fall short.
- New structured sources. Places covers more than 207 million businesses and locations. A Commerce source with more than 40 billion product offers is in private preview. Health questions get ranking that favours clinically credible sources.
- Foundry switch. A Microsoft developer post from Ignite says Web IQ now powers the web search tool in the Foundry Responses API, replacing Grounding with Bing.
- Scopes for buyers. Customers can restrict grounding to approved sources or exclude sources, with up to 10,000 include and 10,000 exclude selectors per scope, according to the same post.
Microsoft also published its own benchmark scores. On FreshSearch, a freshness test it has open-sourced, Microsoft scored Web IQ Auto at 87.5 out of 100 and Perplexity at 57.9. These are vendor figures we could not reproduce.
The crawler AI blockers leave alone
Bing's blog says plainly that robots.txt controls Bingbot's access and that content Bingbot does not crawl stays out of the Web IQ index. Our crawl shows how rarely sites use that control, even when they block every other AI crawler.
The pattern for Bingbot mirrors Googlebot, not the AI bots. Fifty-seven files name Bingbot directly, and 56 of them leave it allowed, usually as part of a search engine allowlist. Of the 21 files that do block it, 19 also block Googlebot, and most belong to ad, tracking and infrastructure domains rather than content sites. Only two block Bingbot while allowing Googlebot: baidu.com and salesforceliveagent.com. Reddit blocks both.
The gap matters most for publishers. Among sites that block all four AI crawlers we tracked, 38 of 59 still admit Bingbot.
| Site | AI crawlers fully blocked (of 4) | Bingbot in robots.txt | Bing tag on homepage |
|---|---|---|---|
| The New York Times | 4 | Allowed | None |
| CNN | 4 | Allowed | None |
| BBC (bbc.com) | 4 | Allowed | NoArchive for Bingbot |
| NPR | 4 | Allowed | NoArchive for all crawlers |
| HuffPost | 4 | Allowed | NoArchive for Bingbot |
| CNET | 4 | Allowed | NoArchive for Bingbot |
| Corriere della Sera | 4 | Allowed | ai-grounding:disallow and ai-training:disallow |
| Reuters | 3 | Named and allowed | None |
| Bloomberg | 3 | Allowed | None |
| Forbes | 3 | Allowed | None |
| The Guardian | 2 | Allowed | NoArchive for Bingbot |
| The Wall Street Journal | 2 | Named and allowed | None |
Source: SaaS Link Building check of robots.txt files and homepage robots meta tags and X-Robots-Tag headers, 18 November 2026. "Fully blocked" means both the site root and a sample article path are disallowed.
Why a robots.txt allow is not the whole answer
Bing deliberately separates crawl access from AI use. In its 30 October post on training and grounding preferences, it set out a page-level ai-grounding value with a disallowextended option that keeps a page available to Microsoft's own products, such as Copilot and Bing's AI answers, while excluding third-party grounding services. The post names Web IQ and grounding in Foundry as those services. Bing has said it will start honouring the new values on 1 February 2027.
Adoption is barely under way. Of 489 unique homepages in our sample, one, Corriere della Sera, carries a Bingbot ai-grounding value, and it chose the full disallow. Robots.txt use of the shared __usage_ai-training token has moved a little: 7 of 508 files now carry it, including Facebook, LinkedIn, WhatsApp, X and IMDb. When we first checked for the NoArchive and NoCache tags Bing is retiring as AI controls on 2 November, we found the token on two sites, though that check used an earlier Tranco list. The training token does not govern grounding, so none of the seven has opted out of Web IQ through robots.txt.
Sites that still send NoArchive or NoCache to Bingbot, such as the BBC, HuffPost and CNET, sit in a transition period. Bing's October post says those tags stop counting as training or grounding preferences after 1 March 2027 unless a site records a temporary grounding preference in Bing Webmaster Tools.
Web IQ's Browse endpoint adds a second route to a page. Its API reference lets a developer return indexed content, crawl a URL when it is missing from the index, or "request a fresh crawl even if indexed content exists". The reference does not name the user agent for those on-demand crawls, though Bing's announcement describes Bingbot as the documented crawler behind Web IQ.
What this means for SEO teams
- Audit your AI crawler policy for consistency. Per-company crawler rules are now common: our October study of AI crawler blocking on 465 top sites found 12% blocking Meta's training crawler alone. If you block GPTBot or ClaudeBot to keep content out of third-party AI products, check whether the same reasoning applies to Web IQ, which serves Bing-indexed content to outside agents.
- Do not block Bingbot to opt out. A robots.txt block removes you from Bing Search and Copilot as well. Bing's grounding preference is the narrower tool.
- Use the extended value if you want Copilot but not resellers. On article templates, send
<meta name="Bingbot" content="ai-grounding:disallowextended">or the headerX-Robots-Tag: Bingbot: ai-grounding:disallowextended. Useai-grounding:disallowonly if you also want out of Microsoft's own AI answers. - Remember grounding has no robots.txt switch. Training can be set site-wide with the shared token, as covered when Google and Bing backed a common training opt-out; grounding must be set page by page.
- If you want agent traffic, check you are in the index. Web IQ results link back to their sources, according to Microsoft. Pages missing from Bing's index can only reach Web IQ through an on-demand Browse crawl, so use IndexNow or a sitemap in Bing Webmaster Tools for new pages.
- Verify Bingbot before trusting it. Bing publishes a Bingbot verification method; use it before writing allow or block rules on user agent alone.
Methodology
On 18 November 2026 we took the top 1,000 domains from the Tranco list (ID V38GN) and requested https://<domain>/robots.txt, then the www host if that failed, with a desktop Chrome user agent. We kept responses with HTTP 200 that were not HTML pages and merged domains that resolved to the same host, leaving 508 unique files. We parsed groups under the grouping rules of RFC 9309: a crawler uses the group that names it, otherwise the wildcard group, with the longest matching rule winning and Allow winning ties. A crawler counts as fully blocked when both / and a sample article path (/news/2026/11/sample-article-page) are disallowed. We checked GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot as AI crawlers, plus Bingbot and Googlebot. We also fetched each domain's homepage, which gave 489 unique pages, and recorded robots, bingbot and msnbot meta tags and X-Robots-Tag headers. Homepage checks miss tags used only on article templates, so they undercount adoption of the grounding value.
Conclusion
Web IQ's general release turns Bing's index into a product that any Azure customer can plug into an agent, and Bingbot is the crawler that feeds it. Our crawl shows most top sites that shut out AI crawlers, publishers included, have not applied the same logic to Bingbot, and with good reason, since blocking it would also cost them Bing and Copilot visibility.
The control built for this case, Bing's ai-grounding:disallowextended value, appeared on none of the homepages we checked, and only one site used any grounding value at all. With Bing due to start reading those values on 1 February 2027, publishers that care where their content ends up have about ten weeks to decide.
Questions and answers
What is Microsoft Web IQ?
Web IQ is Microsoft's search and grounding service for AI agents. It returns web, news, image, video, places and weather evidence with citations from Microsoft's own index. Microsoft made it available to all Azure customers on 17 November 2026.
Which crawler feeds Web IQ?
Bingbot. Microsoft's announcement says Bingbot supports Bing Search, Copilot and Web IQ, and that content Bingbot is disallowed from crawling does not enter the Web IQ index.
How can a site stay in Copilot but out of Web IQ?
Bing's 30 October preferences add a page-level ai-grounding:disallowextended value, sent in a Bingbot meta tag or X-Robots-Tag header. It keeps pages available to Microsoft-operated AI experiences while excluding third-party grounding services, which Bing names as Web IQ and grounding in Foundry. Bing says it begins honouring the new preferences on 1 February 2027.
Can I opt out of AI grounding in robots.txt?
No. Bing says there is no site-wide robots.txt grounding preference. Grounding is set per page. Robots.txt can only block Bingbot entirely, which also removes pages from Bing Search, or set AI training preferences with the shared __usage_ai-training token.
More releases
-
Updates
Google Retires Its Data Feed Validator; All 18 Book Actions Translations Still Link It
Google removed its retired Data Feed validation tool from the English book actions documentation on 18 November, leaving two manual checks and no replacement. Our check found all 18 translated versions still link to the tool, which now returns a 404, and still carry a 20 February update stamp.
-
Studies
Optimizely Hack Hit 178 Brands; 123 of 3,009 Top Sites Run a CSP That Would Block It
An attacker used an Optimizely signup flaw on 16 November to plant malicious JavaScript in customer snippets served on 178 organisations' sites. Report URI found a strict Content Security Policy stopped it. Our crawl of 3,009 top homepages found only 123 with a policy that blocks both routes the payload used.
-
Studies
Only 6.9% of ChatGPT Prompts Show Buying Intent; Our Test Found Brands in 15 of 15
Only 6.9% of prompts in a SparkToro panel of 12,321 US ChatGPT users showed purchase research. Our own 30-prompt test of OpenAI's GPT-6.1 Sol found the other side: all 15 buying questions got brand picks, usually three with one favourite, while 14 of 15 everyday tasks named no brand.
Tell us the site and the goals.
Rankings in Google and visibility in AI answers, from one monthly plan. One call is enough to say which one fits. Email works just as well.