Index your site with the website crawler
Use the Pro website crawler to index your site from its sitemap, by hand or on a schedule, and exclude pages you don't want.
The website crawler fills a knowledge base from your sitemap instead of from hand-picked items. It is the quickest way to give the assistant your whole site, and a crawl schedule keeps it up to date without your help.
The website crawler is part of Promptor Pro. The Crawler tab doesn't appear in the free version. Compare plans.
Before you start
- You need an OpenAI API key under Settings › AI Behavior. The crawler uses it to index the pages. See Get and add an OpenAI API key.
- Your site needs an XML sitemap. WordPress creates one at
/wp-sitemap.xml, and SEO plugins usually create/sitemap.xmlor/sitemap_index.xml. Sitemap index files that point to several sitemaps are supported. - The crawler only reads pages on your own domain. It identifies itself as
PromptorBot/1.0.
Create a separate knowledge base for crawled content, for example site, and keep hand-picked content in another. See Create and manage knowledge bases.
Configure the crawler
- Click Knowledge Bases in the top navigation.
- Open the Crawler tab.
- Under Crawler Configuration, set the options below.
- Click Save Changes.
| Setting | What it does | Default |
|---|---|---|
| Crawl Depth | Shown as how many levels of links to follow from the start URL. See the note below. | 3 |
| Max Pages to Crawl | The maximum number of pages to crawl in a single run. | 100 |
| Crawl Schedule | How often the crawler runs by itself: Never (Manual Only), Hourly, Twice Daily, Daily or Weekly. See Crawl on a schedule below. | Never (Manual Only) |
| Scheduled Crawl Target | The knowledge base and the sitemap URL the scheduled crawl uses. Starting a manual crawl updates both. | Default, /sitemap.xml |
| Content CSS Selector | The part of each page to read, for example .entry-content or #main-article. Use a simple class (.name) or ID (#name). If empty, Promptor tries to find the main content by itself. | .post-content |
| Allowed URL Patterns | Shown as the URLs to include. See the note below. | /blog/.* and /products/.* |
| Disallowed URL Patterns | URLs containing any of these texts are skipped. One per line. | /cart/, /my-account/, /checkout/ |
The crawler takes its list of pages from your sitemap. It applies Max Pages to Crawl and Disallowed URL Patterns to that list. In 2.1.0, Crawl Depth and Allowed URL Patterns don't narrow the list, so every sitemap URL that isn't disallowed is crawled, up to the page limit. To keep pages out of the knowledge base, use Disallowed URL Patterns.
Choosing a content selector
The selector keeps menus, footers and sidebars out of the index, so answers stay focused. To find the right one:
- Open a typical page on your site.
- Right-click the main text and choose Inspect.
- Look for the element that wraps the article text, for example
<div class="entry-content">. - Enter it with a dot for a class (
.entry-content) or a hash for an ID (#content).
If your theme doesn't use .post-content, change the default or clear the field to use automatic detection.
Excluding pages
Add a line to Disallowed URL Patterns for each part of a URL you want to skip. Each line is plain text, not a wildcard. For example, /tag/ skips every URL that contains /tag/.
Good candidates: cart, checkout and account pages, tag and author archives, thank-you pages and legal pages that don't help visitors.
Run a crawl
- On the Manage tab, select the knowledge base the crawler should fill. Use the radio button on its card.
- Open the Crawler tab.
- Under Crawler Initiation, check Sitemap URL. It defaults to
https://your-site.com/sitemap.xml. Change it if your sitemap lives elsewhere. - Click Start Crawling and Indexing.
Promptor reads the sitemap and indexes the pages one by one. Follow the progress in the Log box. When it's done, the knowledge base's card shows the Crawler badge and the new Indexed Docs / Chunks count. The card's Manage button is greyed out, because the crawler manages this knowledge base's content.
To use it in the chat, add the Promptor Chat block and pick this knowledge base, add its shortcode to a page, or select it as the popup's knowledge base. See Add the chat with the Gutenberg block and Add the chat to your site.
Crawl on a schedule
A schedule re-crawls your sitemap by itself, so new and changed pages reach the assistant without any clicks.

- On the Crawler tab, set Crawl Schedule to Hourly, Twice Daily, Daily or Weekly.
- Under Scheduled Crawl Target, choose the knowledge base to fill and check the sitemap URL.
- Click Save Changes.
Under the schedule you now see:
| Item | What it shows |
|---|---|
| Next run: | When the next crawl starts, for example "November 7, 2026 9:52 am (in 24 hours)". While a crawl is running it reads "In progress". With no schedule it reads "Not scheduled". |
| Last run: | When the last scheduled or Run now crawl started, or "Never". |
| Run now | Starts the scheduled crawl immediately. You see "Crawl started. Remaining pages are indexed in the background by WP-Cron." |
The first scheduled run happens one interval after you save. Pages are indexed in the background in batches of 10, so a large sitemap takes a few rounds to finish. Only one crawl runs at a time; clicking Run now during a crawl shows "A crawl is already in progress."
Scheduled crawls use WP-Cron, which runs when your site gets visits. If your site sets DISABLE_WP_CRON, the screen warns you, and a system cron job must call wp-cron.php, otherwise scheduled crawls won't run.
To stop scheduled crawls, set Crawl Schedule to Never (Manual Only) and save. Deactivating Promptor also clears the schedule.
Keep crawled content up to date
With a schedule, crawled pages are refreshed at every run. After a big change, such as a redesign or a new section, click Run now or start a crawl by hand so you don't have to wait for the next run.
Use the crawler, not Sync, to refresh a crawler knowledge base. Sync works from the items you select by hand on the content screen, so on a crawler knowledge base it removes the crawled pages.
Crawl errors
| Message | What to do |
|---|---|
| "Only the current site domain is allowed for crawling." | The sitemap URL is on a different domain, or www doesn't match your site address. Use the exact domain from Settings › General in WordPress. |
| "Could not fetch sitemap…" | Open the sitemap URL in your browser. If it doesn't load, fix the URL. A security plugin or firewall may also be blocking the request. |
| "The provided URL does not contain valid XML…" | The URL points to a normal page, not a sitemap. Use the XML sitemap's address. |
| "Sitemap URL is required." | Enter a sitemap URL. |
| "No URLs found in the sitemap after applying filters…" | The sitemap is empty, or Disallowed URL Patterns excluded every URL. Check both. |
Related
Still need help?
Our support team answers every ticket. Tell us your WordPress version, Promptor version and what you see.
