AI tools like ChatGPT, Claude, Gemini and Perplexity are becoming a new kind of website visitor. Your site needs to be easy for them to crawl, read and understand.
The goal: make your site more likely to show up in AI-generated answers, recommendations and summaries.
Why it matters:
If AI crawlers are blocked, your site may not be readable by tools like ChatGPT, Claude or Perplexity.
Check for:
| Item | What to Review |
|---|---|
| robots.txt | Make sure AI bots are not blocked |
| CDN settings | Check Cloudflare or other security settings |
| Server logs | Look for crawler activity |
| Known bots | GPTBot, ChatGPT-User, ClaudeBot, Claude-Web, PerplexityBot, Google-Extended |
Copy/paste prompt:
Audit this project for anything blocking AI crawlers. Check robots.txt and CDN config for rules blocking GPTBot, ChatGPT-User, ClaudeBot, Claude-Web, PerplexityBot, and Google-Extended. List what is blocked. Do not change anything yet.
Why it matters:
JSON-LD helps machines understand what each page actually is.
Instead of making AI guess, you tell it:
“This is a product.”
“This is an article.”
“This is an FAQ.”
“This is our company information.”
| Page Type | Schema Type |
|---|---|
| Homepage | Organization |
| Blog post | Article |
| Product page | Product |
| FAQ section | FAQPage |
| Local business page | LocalBusiness |
| Event page | Event |