ILC Launches With Yahoo Scout; Only 4 of 73 Supporter Sites Post an RSL License

Ras Al Khaimah, United Arab Emirates, 13 November 2026.

The Internet Licensing Collective (ILC), the nonprofit formerly known as the RSL Collective, launched on 12 November 2026 with Yahoo Scout, Yahoo's AI answer engine, as its first paying licensee. The group says more than 3,000 media brands back it. To see how far that backing reaches into the technical layer AI crawlers actually read, we fetched the robots.txt files of 81 sites the ILC lists as supporters. Only 4 of the 73 files we could read carried the RSL License line that the standard uses to point crawlers to machine-readable terms.

Key findings

• Of 81 ILC supporter domains we checked on 13 November, 73 served a readable robots.txt file; 4 of those 73 (5.5%) included an RSL License directive: The Guardian, Medium, Daily Kos and Drugs.com.
• Only one of the four license files, Daily Kos, returned RSL XML to our fetcher. The other three answered with a bot challenge page or a 403 error.
• None of the 44 supporter homepages that loaded for us declared RSL through an HTML link element or an HTTP Link header, the two other page-level discovery routes in the spec.
• The same supporters block AI crawlers far more often than they publish license terms: 51 of the 73 files fully disallow Common Crawl's CCBot and 46 fully disallow Anthropic's ClaudeBot.
• Across the 1,000 most popular domains on the Tranco list, 479 served a readable robots.txt and 3 carried an RSL license line.

What the ILC launched

According to the ILC's launch release, the collective sells AI companies a single blanket license covering content from its members, and handles verification, usage reporting and royalty distribution. Licensed uses include training, retrieval-augmented generation, grounding, agentic applications and AI-generated overviews. Publishers join free, choose which licenses to take part in, and can still sign direct deals with AI companies.

The release says the ILC runs at cost, with operating expenses projected at or below 10% and the rest paid to rights holders. Royalties depend on how content is used, the economic value of the AI product and how often a publisher's content is relied on. Eric Feng, SVP and GM of Yahoo Research Group, said in the release that "AI answers are only as good as the sources behind them."

The money side is still opaque. Digiday reported that Yahoo pays a blanket fee tied to "demand-side revenue", that co-founder Doug Leeds declined to give the amount, and that each AI company's payments go into a separate royalty pool. Leeds told Digiday that allocation depends on whether content is used for grounding, retrieval or display, and on whether the publisher is credited. Yahoo is so far the only licensee; the ILC website now lists 3,100 brands and more than 3.1 billion licensable works.

Where RSL fits in robots.txt

The ILC co-authored Really Simple Licensing (RSL), an open standard for declaring how AI systems may use content. The RSL 1.0 specification, published on 10 December 2025, gives four equivalent ways to point a crawler to a license file: a License: line in robots.txt, an HTTP Link header, an HTML link element with the type application/rsl+xml, and RSS. The license file itself is XML that lists permitted or prohibited uses such as ai-train, ai-input or search, plus any payment terms.

Membership in the collective and the on-site signal are separate things. A publisher can sign up through the ILC portal without changing its robots.txt, and Leeds told Digiday that "thousands" of publishers use the standard. Our check measures only the public, machine-readable part that any crawler can see.

What we found on supporter sites

The four supporter files with a license line, and what each license address returned to a standard fetch:

Site	License address in robots.txt	What the address returned	Agents given Disallow: / in the file
The Guardian	theguardian.com/license.xml	Bot challenge page (HTML)	37, including ClaudeBot and PerplexityBot
Medium	medium.com/license.xml	Bot challenge page (HTML)	8, including GPTBot and ClaudeBot (a few pages allowed)
Daily Kos	dailykos.com/license.xml	RSL XML: permits search, plus a licensing server entry	9, including GPTBot and CCBot, with /license.xml explicitly allowed
Drugs.com	drugs.com/license.xml	403 Access Denied	20, including ClaudeBot and CCBot

Daily Kos is the only one of the four that adds an Allow: /license.xml exception to its AI-crawler block, and it is the only one whose file we could read. A license file that sits behind bot protection is a practical problem: the crawlers it is written for may never see the terms. We fetched with a generic browser-style user agent, so a crawler on an allow list could get a different answer, but the result shows how easily the file can be caught by the same defences that block scrapers.

The pattern is clear. Blocking remains the default tool: Apple's training token Applebot-Extended is fully disallowed in 54 files, Google-Extended and GPTBot in 30 each. Search-facing agents are treated more gently, with OpenAI's OAI-SearchBot blocked in 14 files and ChatGPT-User in 13. That fits the ILC's own pitch, which presents RSL as working alongside opt-out blocking rather than replacing it.

The wider web: 3 in 1,000

Among the 1,000 most popular domains on the Tranco research list, 479 returned a readable robots.txt (many of the rest are CDNs, ad servers and API hosts with no file). Three carried an RSL license line: The Guardian, Medium and the German tabloid Bild, whose file prohibits both ai-train and ai-input. A fourth, UNESCO, has a License line, but it points to a terms-of-use web page rather than an RSL file. Blocking is far more common: 87 of the 479 fully blocked GPTBot. By contrast, we found llms.txt on 20.8% of top sites in October, a file with no payment terms at all.

Context: licensing signals are multiplying

RSL is one of several machine-readable signals now competing for space in robots.txt. Google noted on 30 October that it intends to support a shared AI training preference signal, and UNESCO's file already uses a Content-Signal line alongside its License line. Our earlier crawl of AI crawler blocking found 12% of 465 top sites blocking Meta's AI training crawler, with none blocking its ads crawler. A paying licensee changes the incentive: if Yahoo Scout payments depend on content being used and credited, a blanket block on the crawlers involved could cost a member money.

What this means for SEO teams

• Decide what you want from AI systems before editing robots.txt. Blocking a crawler, licensing content and allowing search use are three different choices, and RSL lets you state them separately.
• If you join the ILC or publish RSL terms, make sure the license file is reachable by crawlers. Test the URL with a plain fetch and exempt it from bot challenges in your CDN or WAF rules.
• Keep search agents separate from training agents. OpenAI says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers, so a blanket block can remove you from AI search as well as training.
• Watch your logs for Yahoo Scout traffic and for any new licensee the ILC signs; payment terms are tied to usage, so measurement matters.

Methodology

On 13 November 2026 we took the supporter names shown on the ILC homepage and mapped 81 of them to a single primary domain where the mapping was unambiguous (for Wikipedia we used en.wikipedia.org). We excluded supporters whose names did not map clearly to one site. For each domain we requested https://domain/robots.txt and the homepage, following redirects, with a 12 to 15 second timeout. A robots.txt counted as readable if it returned HTTP 200 and contained at least one User-agent line; 73 of 81 did. A site counted as carrying RSL if any line began with License:. We then fetched each license address. A crawler counted as fully blocked when a group naming it contained Disallow: / without an allow-all rule; wildcard (*) groups were not counted. We checked homepage HTML (first 400 KB) and response headers for the application/rsl+xml type; 44 homepages loaded. We ran the same robots.txt checks on the top 1,000 domains of the Tranco list downloaded on 13 November 2026. Bot protection blocked some requests, so counts are a floor, not a ceiling.

Conclusion

The ILC now has a product, a paying customer and thousands of backers, but on the open web its licensing signal is still rare. Most supporters we checked continue to rely on robots.txt blocks, and three of the four license files we found could not be read by a standard fetch. If collective licensing is going to pay publishers based on usage, the next step is making those terms visible to the machines that are supposed to honour them.

Questions and answers

What is the Internet Licensing Collective?

The Internet Licensing Collective (ILC) is a nonprofit, publisher-led collective rights organization, formerly the RSL Collective. It sells AI companies one blanket license to its members' content and distributes royalties. It launched on 12 November 2026.

Who is the ILC's first licensee?

Yahoo Scout, Yahoo's AI answer engine, is the ILC's first confirmed commercial licensee. Yahoo pays a blanket fee whose amount has not been disclosed.

How do publishers get paid by the ILC?

According to the ILC, royalties depend on how content is used, the economic value of the AI product and how often a publisher's content is relied on. Operating expenses are projected at or below 10%, with the rest going to rights holders.

How does a site declare an RSL license?

The RSL 1.0 specification allows a License: line in robots.txt pointing to an RSL XML file, an HTTP Link header, an HTML link element with the type application/rsl+xml, or RSS. All four methods are treated as equivalent.

How many ILC supporter sites publish an RSL License line?

In our 13 November 2026 check, 4 of 73 ILC supporter sites with a readable robots.txt carried an RSL License line: The Guardian, Medium, Daily Kos and Drugs.com.

###