AI engines like ChatGPT learn from the internet. They read your website so they can answer users' questions. You can decide whether to give them access or block them. You do it with a simple text file called robots.txt, which holds instructions for crawlers.
What is a robots.txt file, anyway?
A robots.txt file is a plain text file that sits in the root folder of your website. It gives instructions to "crawlers" - automated programs that scan the internet. Think of it like a "do not disturb" sign you hang on certain doors in your house (your website).
The crawlers of Google, ChatGPT and other tools check this file before they start crawling your site. That's how they know which pages they're allowed into and which they aren't. It's an accepted standard, and most well-known crawlers respect the instructions in it.
Diagram
How a crawler reads your website
What happens every time a crawler arrives at your site
The crawler arrives
For example, OpenAI's GPTBot arrives at your address.
It checks robots.txt
It looks for the file at yoursite.com/robots.txt.
It reads the instructions
The file tells it which pages or folders it may not enter.
It follows the instructions
It crawls only the allowed pages and skips the blocked ones.
Why would I want to block AI engines?
There are two main reasons to block AI crawlers. The first is to protect your original content. The second is to control the information they present about you, which can sometimes be inaccurate or out of context.
For example, let's say you're a baby sleep consultant and you've written a detailed, unique guide on your website. If you let ChatGPT read it, it may summarize your method and present it to other users as its own answer, without giving you credit and without those users ever reaching your site. Blocking stops your content from becoming fuel for AI engines.
Diagram
Allow or block AI access?
The main considerations for each choice
| Attribute | Why block | Why allow |
|---|---|---|
| Protecting content | Prevents copying and use of your original content without credit. | Your content won't be part of the answers, not even with a link. |
| Controlling your image | Stops AI from presenting partial or wrong information about your business. | Your business won't appear in AI recommendations and summaries. |
| Traffic to your site | People have to visit your site to read the full content. | Can bring new traffic from people who get an AI recommendation. |
| Copyright | Prevents your work (images, text) from being used to train models. | AI won't learn about your expertise straight from the source. |
Protecting content
Why block
Prevents copying and use of your original content without credit.
Why allow
Your content won't be part of the answers, not even with a link.
Controlling your image
Why block
Stops AI from presenting partial or wrong information about your business.
Why allow
Your business won't appear in AI recommendations and summaries.
Traffic to your site
Why block
People have to visit your site to read the full content.
Why allow
Can bring new traffic from people who get an AI recommendation.
Copyright
Why block
Prevents your work (images, text) from being used to train models.
Why allow
AI won't learn about your expertise straight from the source.
Why would I want to give AI engines access?
The other side of the coin is exposure. More and more people use AI engines like Perplexity, or Google's built-in AI search, to get answers and recommendations. If you block them, you simply won't exist in that world.
Let's say you run a boutique bakery in Kfar Saba. Someone asks Perplexity: "Where can I get a good almond croissant in Kfar Saba?" If Perplexity crawled your website and read your menu and reviews, it may recommend you and send you a new customer. If you blocked it, it will skip you and recommend your competitors.
What it looks like on screen
An AI answer that brings you customers
An example of an answer in an AI search engine such as Perplexity
Where's a good boutique bakery in the Sharon area with vegan options?
The 'Bread & Friends' bakery in Moshav Tzofit offers a wide range of handmade pastries, including vegan croissants and yeast cakes. According to their website, they use local ingredients and everything is baked on site every morning.
Source: lechem-friends.co.il
An illustration of the screen, drawn in code. The real interface changes from time to time.
What are AI crawlers called?
To block or allow access, you need to know the crawler's exact name. That name is called the User-Agent. Each company has its own name for its crawler, and here are the three most important right now:
- GPTBot: OpenAI's crawler - the company that created ChatGPT. It collects information to train their future language models.
- Google-Extended: a dedicated Google crawler. It collects information for AI products like Gemini (formerly Bard) and AI features in search. Important to understand: blocking it doesn't hurt your regular Google ranking, because Google's main crawler is called
Googlebotand it works separately. - PerplexityBot: the crawler of the Perplexity AI search engine, which gives direct answers to questions with cited sources.
Diagram
Which crawlers should you know?
These are the names you'll use in your robots.txt file
Yes, like this
- GPTBot (OpenAI, makers of ChatGPT)
- Google-Extended (Google, for Gemini)
- PerplexityBot (Perplexity AI)
Not like this
- ChatGPT (that's the product name, not the crawler)
- Google (that's the company name, not the crawler)
- Gemini (the product name, not the crawler)
How do I block all the AI crawlers at once?
To block all three crawlers we mentioned, create a robots.txt file and add the text below. This code tells each of these crawlers separately that it may not access any page on the site.
Here's the exact code to copy:
Code
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: PerplexityBot
Disallow: /Let's break it down:
User-agent: GPTBot– addresses OpenAI's crawler specifically.Disallow: /– tells it "you may not go anywhere". The/stands for the whole site, from the homepage down.- The blank line separates the instructions for different crawlers.
How do I block only some of them, or only part of my site?
You don't have to block everything. You can be more specific. For example, you might want to let Google in so you appear in its results, but block OpenAI so it doesn't train its models on your content.
In that case, you can use this code:
Code
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Allow: /
User-agent: PerplexityBot
Disallow: /private-files/Here's what this code does:
- Blocks GPTBot completely from the whole site (
Disallow: /). - Explicitly allows Google (Google-Extended) access to the whole site (
Allow: /). - Blocks PerplexityBot from one specific folder called
private-files. It can still crawl the rest of the site.
How do I add a robots.txt file to my website?
Adding the file is a fairly simple process. You create a text file and upload it to your server. If I built your website, I can do it for you as part of the monthly support service, starting at ₪500 a month.
If you'd like to do it yourself, here are the steps:
Diagram
How to add a robots.txt file to your site
Four simple steps to set instructions for crawlers
Create a text file
Open a simple text editor (like Notepad on Windows or TextEdit on a Mac) and create a new file.
Copy the instructions
Copy and paste the blocking or allowing rules you chose into the file.
Save the file
Save the file with the exact name 'robots.txt' (lowercase, no quotes).
Upload to the server
Upload the file to your website's root folder. The final address should be yoursite.com/robots.txt.
If you have a WordPress site with an SEO plugin like Yoast or Rank Math, there's usually a settings area that lets you edit the robots.txt file directly, with no need to upload a file.
What to do now?
- Decide on your strategy: is protecting your content or getting exposure in AI engines more important to you? There's no single right answer - it depends on your business.
- Check whether you already have a file: go to
your-domain.com/robots.txt. If you see text, you have an existing file. If you get a 404 error, you don't. - Create or edit the file: use the code in this article to create a new file or update your existing one with the right instructions.
- Upload the file to your site: make sure it's in the right location and that you can open it in a browser.
- If it sounds complicated, don't struggle with it. I can help you set it up properly. Get in touch through the contact page.
