About the crawler

Who it is

Requests identify themselves as mock-data.paulcheek.com research bot with a link back to this page.

What it does

When somebody enters a company domain, the crawler reads a bounded set of that company’s public pages to understand what the business does: the homepage, about, products, pricing, customers, press, careers and leadership pages, plus whatever the sitemap points at. It stops at roughly forty five pages, two links deep, and it does not submit forms, follow logouts or attempt authentication.

What it respects

• robots.txt disallow rules for user-agent *.
• A hard page cap and depth cap per company.
• A small fixed concurrency, so it never behaves like a load test.
• Content hashes, so a refresh re-reads only pages that actually changed.

What it stores

Page text, title and URL, once per company domain rather than once per user. That text becomes a knowledge graph and a search index. It is public information, kept so the same pages are not fetched repeatedly.

How to exclude your site

A robots.txt disallow is honoured immediately. If you would rather your company were excluded from the product entirely, including from anyone at your own company generating a mock instance, write to paulcheek.com/contact and we will block the domain.