📖 Overview

📍 The web crawler will scan only web pages from your domain. Links to content on other domains or documents/PDFs will be ignored.

📄 We recommend using a URL dedicated to an FAQ, knowledgebase or help center, where the content primarily focuses on answering customer questions. Including irrelevant content may decrease the AI Chatbot's success rate.

⏲️ The scan process can take up to 15 minutes, depending on the amount and type of the website.

Getting there

You can access "scan a website" via Content management page

Or: Go to Settings → AI Chatbot → Knowledge Center → Click Add new → Website

🚀 Quick start

There are 3 ways to scan a website

What is a Sitemap?

A sitemap is a file that lists all the website’s pages to help search engines scan the site better, and it can usually be accessed by adding/sitemap.xml to the website’s domain - for example, google.com/sitemap.xml.

1. Fire a scan: Scan max 100 pages from a website (multi page scan)

Enter a URL, select "Website", Click "Save".

Your scan is now in progress - it will might take a few minutes. A result notification will appear on screen after processing - succuss or failed result of the scan. If the scan failed or the pages scanned do not contain some critical content of the page - we will try again by tweaking some of the advanced settings.

2. The scan failed or the extracted content was insufficient?

Here's some rules of thumb to extract pinpoint results 👍

Here's some helpful rules of thumb - these will help you extract the best results possible:

✅ If crawl has failed and "Proxy type" is "Datacenter" - the scan was most likely was blocked. Switch it to "Residential". Info about proxy type

✅ If a webpage has some loading state or you are aware the page is rendered dynamically by Javascript - use Crawler type: Dynamic

✅ If the extract content is missing some crucial text that does exists on the page - turn the HTML transformer OFF

Scan settings

Scan mode

Screenshot - Crawl type

• Specific page: scan a single page. example: enter the URL - https://docs.glassix.com/docs/bot-best-practices, we’ll scan only that particular page.
• Website:: Scan the URL and links found on that pages and the next ones. The number of pages scanned is controlled by max results and max depth properties or by sitemap.

General Advanced settings

Crawler type

• Static: if your webpage is based on classic HTML, CSS, JS combo.
• Dynamic If there is a visible page loading or your website is based on SPA (single page application - such as React, Angular, Vue)

👍 Rule of thumb: is there any loading visible in the webpage? - use "Dynamic".

Proxy type

• Datacenter: we will scan your website from these 3 IP's:
  107.175.80.80, 162.212.175.22, 107.172.69.119. This way - you can whitelist these IP's to ensure a successful scan of your website.
• Residential: the scan will be initiated using residential IP addresses in your current country.

👍 Rule of thumb: If a scan failed using "Datacenter" switch to Residential. This will bypass blocking mechanisms of websites.

CSS selectors to click on

Use this setting to specify which elements on the page the crawler should interact with. For example, elements like buttons, links, or dropdowns can be targeted. You can define selectors using attributes like aria-expanded="false" to target collapsible sections, class*="question", class*="FAQ", class*="accordion", and similar patterns.

CSS selectors to ignore

This setting allows you to exclude specific elements from the crawl, such as navigation bars, footers, and scripts.

Text to Ignore

Text snippets that will be removed from the extracted results.

HTML Transformer

When active - the scan will extract what seems to be only the relevant text content, by removing small unmeaningful text.

👍 If content appears on the webpage but is missing from the scan, turn **off ** the HTML transformer.

Multi page advanced settings

Max crawl depth

This setting controls how deep the crawler should go within your site’s link structure. For example: if set to 5 - the crawler will follow links up to 5 levels deep.

Max pages

This setting limits the number of pages the crawler will scan. Max 3000 pages.

Use sitemap

Enable this option if you want the crawler to prioritize scanning URLs listed in your website's sitemap.

URL globs to include

Specify patterns or globs to include in the crawl. For example, you can use /products/** to ensure that only product pages or a specific section of your website are crawled. This helps focus the crawl on relevant parts of the site.

You can test URL globs here

URL globs to ignore

You can define globs here to exclude certain URLs from the crawl, such as specific sections or file paths that don’t need to be scanned. For instance, if /products/** or certain directories should be skipped, add those here.

You can test URL globs here

🔦 Examples

We will try to match the optimal scan settings for each scenario

FAQ