The SEO Knowledgebase, searchable & sorted
Clear, jargon-free guides on technical, local and content SEO — the practical foundations and field-tested checklists behind every page-one ranking your business is chasing.
Robots.txt
Robots.txt is a plain text file named robots.txt at the root of a site host. It tells cooperative crawlers which URL paths they may request. It controls …
PerplexityBot
PerplexityBot is the declared crawler from Perplexity. It fetches and indexes public webpages so Perplexity can surface and link websites in search results, …
Schema Markup
Schema markup is structured code that labels webpage facts for machines. It uses Schema.org vocabulary to describe entities, Types, and Properties. Schema.org …
llms.txt
llms.txt is a proposed Markdown file that lists selected public website resources for AI tools and agents. It points tools toward chosen pages, documentation, …
Source Citation
Source citation is the visible link an AI answer shows beside a claim. It points readers to a webpage, document, or record that may support the answer. A …
Claude-SearchBot
Claude-SearchBot is the crawler from Anthropic for Claude search. It navigates public webpages to improve search result quality. This page covers bot …
GPTBot
GPTBot is OpenAI’s web crawler for content that may be used to train generative AI foundation models. You can control GPTBot with robots.txt, but GPTBot is …
OAI-SearchBot
OAI-SearchBot is the OpenAI crawler for ChatGPT Search. The crawler requests public webpages so ChatGPT Search can surface websites in search results. …
AI Crawler
An AI crawler is an automated web client that fetches public URLs for an AI product. AI crawlers can support search answers, source retrieval, model training, …
Can’t find the answer? Get a free, no-pitch SEO audit.
Our team reads every page like Google does and sends back the exact technical, content and link fixes holding your site back — usually within 48 hours.