Skip to main content
SupaCortex
Sign in Book a Demo
Feature

Your help centre, read again every day

Help articles change every release. Instead of uploading them again, give SupaCortex the address once - it re-reads the site on a schedule, fetches only what changed and keeps answers in step with what is published.

Book a demo See pricing
  • TheDocs & Google Sites presets
  • Only changed pages fetched
  • A broken crawl never wipes a project

How does the SupaCortex website crawler work?

A web-crawler feed reads a public site - a help centre, a Google Site or any docs site - from its sitemap or by following links from the start pages you give it. Each page is cleaned of menus, headers and footers, kept with its headings, and indexed as a document that links back to the live page. The feed runs again on a schedule: unchanged pages are skipped, changed ones are re-indexed, and pages that have gone are removed only after a complete, healthy run.

  • Presets for TheDocs help centres and Google Sites; works on any public docs site.
  • Sitemaps, include and exclude patterns, page and depth limits.
  • Respects robots.txt and noindex; waits between requests.
  • Only changed pages are fetched again; Run now for an immediate refresh.
  • Amazon S3 and S3-compatible buckets as a second feed type.
  • Category, tags, trust tier and customer ID set once for the whole feed.
How it works

Adding a site

  1. 01

    Give it the address

    A start page or a sitemap, and a preset if it is a TheDocs help centre or a Google Site.

  2. 02

    Set the boundaries

    Allowed hosts, paths to include or skip, and a page limit - up to 5,000 pages per feed.

  3. 03

    Choose a schedule

    Daily, every few hours or hourly, depending on your plan. Run now is always there.

  4. 04

    Check the run

    Each run lists pages seen, fetched, unchanged, failed and removed, with errors you can act on.

In detail

Built to stay correct over months, not just on day one

Presets for the sites you already have

A help centre's article pages are what matter; its category listings, search and login pages are noise.

  • TheDocs: reads the sitemap and the article body, and skips home, category, series, tag and search pages.
  • TheDocs knowledge bases can also use a dedicated feed that reads each article's Markdown directly.
  • Google Sites: reads each page's content sections and stays inside the one published site.
  • Generic: any public site, with an optional CSS selector for the part of the page that holds the content.

Safe when the source misbehaves

A crawler that deletes everything when a site returns errors for an hour is worse than no crawler.

  • Pages missing from a source are removed only after a run that completed and saw at least half the feed.
  • Unchanged pages are detected by fingerprint and server headers, and are not fetched or re-indexed.
  • Private, internal and loopback addresses are refused, checked again after every redirect.
  • Pages over 10 MB, slow responses and endless redirects are cut off rather than retried forever.

Files in S3 buckets

Some knowledge is not a website at all - exports, manuals and generated reports sitting in a bucket.

  • Amazon S3 or an S3-compatible store, by bucket and prefix.
  • Access keys stored encrypted and never shown again; a public bucket needs none.
  • The same file types as uploads: PDF, DOCX, XLSX, Markdown, text and HTML.
  • Objects removed from the bucket are removed from the project by the same safe rule.

Related

Google Drive, SharePoint & Confluence

Connect once with OAuth, pick folders, sites or spaces, and only changed files are fetched again.

Document Uploads

Drop in PDFs, Word and Excel files, Markdown and HTML in bulk - or a zip of them. Each is parsed and indexed.

AI Support Assistant for Your Website

Answer customers' how-to questions on your site, any hour, from the docs you already have.

SaaS & Software

Docs that change every release, answered correctly the day they change.

Integrations

Every source it learns from and every place it answers, with what is live and what is next.

SaaS: Half the How-To Tickets Answered on the Site

A widget on the help centre, fed by the help centre, with every answer cited.
Frequently asked

Questions people ask before they start

Any public website, with presets for TheDocs help centres and Google Sites. When a site has a sitemap, the sitemap decides which pages are read; otherwise the crawler follows links from the start pages you give it, up to a depth and a page limit you set.

On the schedule you choose, and whenever someone presses Run now. The fastest schedule depends on the plan: daily on Starter, every six hours on Growth, and hourly on Business and Enterprise.

Its document is removed after a complete run that no longer finds it. A run that failed, or that saw less than half of the feed's pages, removes nothing - so a site that is down for an hour cannot empty your project.

Not with the web crawler, which reads public pages only. For a TheDocs knowledge base, the dedicated TheDocs feed can also read private articles with an API key - but anything indexed into a project can be answered for everyone who can ask that project, so use it only on a project whose whole audience may read those articles.

Yes, by default. It identifies itself as SupaCortexBot, honours robots.txt rules and crawl delays, skips pages marked noindex, and waits between requests.

Yes. An S3 feed reads a bucket and prefix on Amazon S3 or an S3-compatible store, picks up the same file types as uploads, and keeps the project in step with the bucket.

Send us your help centre's address. We will show you it answering.

Book a 30-minute demo on your own documents. We set up a 14-day trial workspace for you - no credit card, no setup fee.

Book a demo See pricing

Questions first? Email info@greatdevelopers.com or call +91 8585953085.

Still curious? Ask your favorite AI about SupaCortex.

ChatGPT Claude Perplexity Grok