Is your hotel website blocking the AI your guests use?


Scroll to read
Skift recently checked the robots.txt files of 50 of the biggest travel sites, the file that tells bots what they can and can’t read. Only 10 said anything at all to AI crawlers. Eight sites, mainly hotels, wouldn’t even load for Skift’s automated checks. They returned a 403 Forbidden error.
At the same time, Adobe reported AI visits to travel sites up 119% year over year. Guests are sending assistants to look at hotels, and a lot of hotel websites are locking the door on them without anyone deciding to.

Most hotels haven’t decided. Their website did.

Very few hotels have sat down and made a call on AI. The decision got made somewhere else: a security plugin, a hosting firewall, a bot setting on the CDN. Those tools were set up to stop scrapers copying your rates. They don’t know the difference between a scraper and an assistant answering a guest who asked for a pet-friendly place near the beach with parking.

Even the big players are still working it out. Skift found Airbnb allowed OpenAI’s and Anthropic’s crawlers in August and blocked both by October. If companies with whole legal teams are changing their minds month to month, a 40-room hotel shouldn’t be deciding this by accident.

Not every AI bot does the same job

This is the part most of the conversation misses. The AI companies run different bots for different jobs:

  • Training crawlers (OpenAI’s GPTBot, Anthropic’s ClaudeBot) collect content to train future models.
  • Search crawlers (OAI-SearchBot, Claude-SearchBot) index pages so an assistant can cite them and link to them in its answers.
  • Live fetchers (ChatGPT-User, Claude-User) load a page right now because a person just asked about it.

You can reasonably say no to training and yes to the other two. Some chains go further: Skift found Best Western and ITC Hotels explicitly welcome OpenAI’s training crawler. Either way, it should be a choice. Blocking the search and live bots means the guest’s assistant can talk about you, but can’t send them to you.

What I’d check this week

  1. Read your robots.txt. Type your domain followed by /robots.txt. If it says nothing about AI, that’s a default, not a decision. If it blocks everything, find out who chose that and why.
  2. Check the firewall, not just the file. A 403 means the door is locked whatever robots.txt says. Ask your web host or IT team whether bot protection is turning away AI user agents.
  3. Make the booking path readable. Room types, rates and policies written as text on the page, not only inside images or behind a script that only loads for humans.
  4. Ask an assistant about yourself. Ask ChatGPT about your hotel. Can it describe your rooms and link to your own site, or does it only link to an OTA?
  5. Write the decision down. What you allow, what you block and why. Review it every quarter, because this is moving every month.

Summing up

Adobe’s Vivek Pandya put it plainly: when AI can’t read a site, “potential revenue is being left on the table.” The fix is rarely a big project. It’s a decision, a short edit to one file and a conversation with whoever runs your firewall.

The OTAs have already made their call. A direct booking only happens if the guest, or the assistant working for them, can actually get to you. Make sure yours can.

Source: Skift, “Dear Robot: How Travel Is (and Isn’t) Talking to AI”, 9 November 2026, including Adobe and Cloudflare data cited there.