United Bars AI Agents by Contract; Only 3 of 41 Airline Robots.txt Files Name AI Bots

United Airlines restricted third-party AI agents on 11 November through its terms, not its robots.txt. Our request to 75 airline websites found that only 3 of the 41 robots.txt files we could read name any AI bot, none blocks user-initiated agents, and 34 sites returned no readable file.

Vibhu Dhariwal
Cofounder & CTO, Digital Gratified
Published
Section
Studies
Reading time
8 minutes
Data card: 3 of 41 airline robots.txt files name any AI bot, from a check of 75 airline websites after United restricted AI agents

United Airlines announced on 11 November that it is restricting third-party AI agents and assistants from its website and app, and will also limit them through travel agencies and booking platforms. The airline did it through its terms and contract of carriage, not its crawler file: when we read United's robots.txt on 15 November, it named no AI crawler or agent at all. To see whether that is typical, we requested the robots.txt files of 75 airline websites. Only 3 of the 41 files we could read name any AI bot, and none of them blocks a user-initiated agent.

Key findings

  • We requested robots.txt from 75 airline domains across five regions and could parse 41 files. The other 34 sites refused the request (13), dropped or timed out the connection (12), served an HTML page in place of the file (5) or returned a 404 (4).
  • 38 of the 41 readable files name no AI crawler, so every AI bot falls under the wildcard User-agent: * group. In all 41 files that group leaves the homepage open to any crawler.
  • Only 3 airlines name AI bots: Alaska Airlines, Hawaiian Airlines and Porter Airlines. Alaska and Hawaiian each name 16 AI user agents: 14 share the rules that apply to every crawler, and only CCBot and Bytespider are shut out. Porter blocks 14 of the AI crawlers on our list, mostly training bots.
  • No readable file disallows a user-initiated agent such as ChatGPT-User, Claude-User or Perplexity-User. The two files that name them do so to grant access.
  • 18 of 41 files already fence off booking, checkout or trip management paths, United among them, with rules covering its fare calendar, reservation and cancellation pages.
Waffle chart of 75 airline websites: 3 name AI bots in robots.txt, 38 leave AI bots to the wildcard group, 13 refused the request, 12 dropped the connection, 5 served an HTML page and 4 returned a 404
Outcome of a robots.txt request to 75 airline domains on 15 November 2026. Source: SaaS Link Building crawl; United's file read through a separate web fetch after our crawler was refused.

What United changed

United's announcement says sharing account credentials with outside AI assistants can expose personal data and lead to unintended purchases, account changes or use of MileagePlus miles. It also argues automated actions can clash with FAA and DOT rules, pointing to check-in, where a person must confirm a bag holds no prohibited items. United said it will block unauthorized personal AI agents on its own channels and restrict them indirectly through updated booking, ticketing and distribution policies for agencies and aggregators.

Search Engine Journal reported that the updated website terms exempt search indexing, assistive technology and ordinary browsers used by a person, while barring the use of fares, schedules and availability data to train or ground AI models. United's legal pages load only with JavaScript, so we could not open those terms directly; the press release and robots.txt are the parts we checked ourselves.

United's robots.txt reads like a classic airline setup. Its wildcard group disallows agency PDFs, the fare calendar, reservation view and cancellation pages and /api/*, and a second group restricts a single legacy crawler, HuaweiSymantecSpider. There is no Sitemap line and no AI-specific group. United's homepage also dropped our crawler's connection, which suggests the airline leans on edge bot management and its terms rather than robots.txt.

The three airlines that name AI bots

AirlineAI user agents namedBlocked from the whole siteApproach
Alaska Airlines16CCBot, BytespiderAI search, assistant and training bots share the wildcard rules; file comment dated 10 November 2026
Hawaiian Airlines16CCBot, BytespiderSame list as Alaska; file comment dated 21 October 2026
Porter Airlines14GPTBot, ClaudeBot, Google-Extended, CCBot and 10 othersCloudflare managed robots.txt with a Content-Signal line set to search=yes, ai-train=no

The Alaska and Hawaiian files are the clearest example of a pro-agent stance in the sample. Both place GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended and others in the same group as User-agent: *, and Hawaiian's file carries a comment saying they are explicitly allowed under the same rules as everyone else. Both then disallow booking and shopping flows in that shared group. That is close to the opposite of United's position.

Porter is the only airline in the sample using Content Signals, the comment-and-directive format Cloudflare offers in its managed robots.txt. Its file allows search, says no to AI training and blocks a long list of training crawlers by name, but it does not name ChatGPT-User, Claude-User or Perplexity-User.

Why robots.txt was never going to stop booking agents

The gap between United's terms and its robots.txt is less odd than it looks. Google's documentation for user-triggered fetchers, which includes Google-Agent for agents that act on a user's request, says these fetchers "generally ignore robots.txt rules" because a person asked for the fetch. OpenAI's crawler documentation says the same of ChatGPT-User: since its actions are started by a user, "robots.txt rules may not apply."

So a robots.txt line aimed at a booking agent would carry little weight, which is why United reached for contracts, cancellations and bot management. We covered how one fast-growing agent identifies itself in our report on Meta's Muse and Web Bot Auth signing keys, and how to spot Google-Agent sessions in server logs.

Unreadable files carry their own search risk. Google's robots.txt specification treats 4xx responses other than 429 as if no robots.txt exists, and treats timeouts and dropped connections as server errors, under which Google stops crawling a site for the first 12 hours while it retries. Our crawler ran from a cloud server and Googlebot is often treated differently by bot managers, so a refusal here does not prove Google sees the same thing. It does mean 26 of 75 airlines, United included, answered our request for the one file meant for crawlers with a 403, a 401 or a failed connection.

Regional breakdown

RegionDomains requestedRobots.txt readableName AI botsFence booking paths
North America141137
Latin America7200
Europe211105
Middle East and Africa10501
Asia Pacific231205
Total7541318

Latin American carriers were the hardest to read: 5 of 7 refused the request, dropped it or served something other than a robots.txt file. Lufthansa, SWISS and Austrian, all Lufthansa Group carriers, each publish a short file that allows everything except deep links and secured service paths, and none names an AI crawler.

What this means for SEO teams

  • Separate the three jobs. Robots.txt can opt out of AI training (GPTBot, Google-Extended, Applebot-Extended) and AI search indexing (OAI-SearchBot, Claude-SearchBot, PerplexityBot). It is a weak tool for user-triggered agents, which the vendors themselves say may not follow it.
  • Keep agent rules out of search rules. United's terms exempt search indexing. If your business restricts agents, check that firewall or bot-management rules do not also catch Googlebot or Bingbot, and verify with the URL Inspection tool in Search Console.
  • Make robots.txt reachable. Test the file from outside your network. A 403 means Google treats the site as having no robots.txt, and timeouts are handled as server errors.
  • Fence transactional paths for everyone. 18 of the airlines in our sample already disallow booking or checkout URLs. That protects crawl budget whatever an agent's policy is.
  • Watch the travel vertical in AI answers. Our earlier test found AI Mode sent most flight links to Google's own pages. Airlines that block agents may push more of that traffic to Google and other intermediaries.

Methodology

On 15 November 2026 we built a list of 75 airline domains covering large carriers and low-cost airlines in North America (14), Latin America (7), Europe (21), the Middle East and Africa (10) and Asia Pacific (23). We requested https://<domain>/robots.txt once per domain with a standard desktop browser user agent over HTTP/1.1, following up to five redirects with a 25-second timeout. A response counted as a robots.txt file if it contained at least one User-agent line and was not an HTML page; one gzip-encoded file (Hawaiian) was decompressed. United's file could not be fetched by our crawler and was read through a separate web fetch on the same day. We did not retry blocked sites with other user agents.

We parsed each file with Python's standard robots.txt parser and a list of 31 AI user agents (training, AI search and user-initiated fetchers). An airline "names" an AI bot when a User-agent line matches one exactly. "Blocked from the whole site" means the parser denied that agent the homepage. Booking-path rules are Disallow lines containing booking, reservation, shopping, checkout, trip management, payment or flight search paths; we checked matches by hand and excluded destination pages and a flight-notification path. Results reflect a single snapshot and our crawler's network position, not what Googlebot or any AI vendor receives.

Conclusion

United's ban on AI agents is a legal and bot-management move, and the airline industry's robots.txt files show why: 38 of 41 readable files do not mention AI at all, and the only carriers that name user-initiated agents, Alaska and Hawaiian, do so to welcome them. For SEO teams the lesson is to treat robots.txt as a training and search-indexing control, and to make sure any agent blocking stops short of the crawlers that send search traffic.

Questions and answers

Did United Airlines block AI bots in its robots.txt file?

No. When we read United's robots.txt on 15 November 2026 it named no AI crawler or agent. United announced its restrictions on third-party AI agents on 11 November through its terms, contract of carriage and bot blocking on its own channels.

How many airline websites name AI bots in robots.txt?

In our 15 November 2026 check of 75 airline domains, 41 returned a readable robots.txt file and 3 of those named any AI bot: Alaska Airlines, Hawaiian Airlines and Porter Airlines.

Do AI agents such as ChatGPT-User follow robots.txt?

Not reliably. OpenAI says robots.txt rules may not apply to ChatGPT-User because its actions are started by a user, and Google says its user-triggered fetchers, including Google-Agent, generally ignore robots.txt rules.

What happens if a robots.txt file returns a 403 error to Google?

Google's robots.txt specification says its crawlers treat 4xx responses other than 429 as if no valid robots.txt file existed. Timeouts and dropped connections are treated as server errors instead.

More releases

  1. Studies

    Reddit Visits Fell 13% in October; Our Repeat Test Found It in 19 of 36 Top 10s, From 30

    Reddit's global web visits fell 13% year on year in October, Similarweb data shows. Our repeat check of 36 US Google searches on 15 November found a Reddit thread in the top 10 on 19, down from 30 on 4 November, while AI Overviews kept citing Reddit at about 6%.

  2. Studies

    Top Million Sites Lost 12% of Visits; 5 Platforms Now Hold 77% of Our Google Estimate

    Similarweb data shared by SparkToro shows browser visits to the top million websites fell 12.2% from Q1 2022 to Q3 2026. Our pull of Google organic estimates for 198 of those domains found the median site flat, while five platforms grew from 69.7% to 77.1% of estimated traffic.

  3. Studies

    Does Google Score Whole Sites? Trial Notes Suggest So; Page Links Won 21 of 24 SERPs

    Google says it has no website authority score, yet its ranking guide cites site-wide signals and a Google engineer's trial notes call its quality score largely site-level. In our pull of 60 live searches and 24 snapshots, page-level link scores tracked rankings far better than domain scores, winning 21 of 24 head to head.

Every release

Tell us the site and the goals.

Rankings in Google and visibility in AI answers, from one monthly plan. One call is enough to say which one fits. Email works just as well.