ChetverBot
the bot that checks websites
ChetverBot is Chetver’s website checker. It reads public pages and files to check their accessibility, content and structure, and shows the results in Chetver.
- User-Agent
ChetverBot/1.0 (+https://chetver.com/bot)- robots.txt
User-agent: ChetverBot
A ChetverBot visit is a Chetver check. It does not mean that ChatGPT, Claude, Gemini or Perplexity visited or recommended your website.
How to recognise ChetverBot
by the signature in the request
ChetverBot sends this User-Agent:
ChetverBot/1.0 (+https://chetver.com/bot)In robots.txt it follows the group User-agent: ChetverBot. If that group is absent, it follows User-agent: *.
ChetverBot runs on Google Cloud. We do not publish a fixed IP list. The User-Agent identifies the claimed crawler, but anyone can copy it; it is not proof of identity.
What it reads and how often
public pages and files only
Every check has its own reason and its own scope. Here’s when ChetverBot comes to your site.
Site readiness
Up to 30 pages of a connected website, after payment, weekly, and when the owner requests another check.
Daily availability checks
The homepage and robots.txt, sitemap.xml and llms.txt for eligible active websites.
AI Page
Up to 30 pages when the owner creates or refreshes it, and automatically every 7 or 30 days depending on the plan.
Important pages
Pages selected by the owner, when the owner requests a check.
Free scan
The homepage, robots.txt and sitemap.xml when someone requests a scan on chetver.com.
ChetverBot reads public responses without running JavaScript, completing forms, signing in or placing orders. It does not change your website.
How it follows robots.txt
and what happens if the file is unavailable
Before reading a page, ChetverBot checks robots.txt and skips disallowed paths. Unchecked readiness checks remain unknown; they do not become a zero score.
Rules are cached for up to 60 minutes. Allow that time for a change to take effect. ChetverBot can fetch robots.txt itself even when other paths are blocked.
The site readiness check stops if robots.txt is unavailable. Other checks may continue without robots rules when the file cannot be loaded; Chetver records that failure.
Chetver also checks the availability of the AI Page it hosts on a customer’s aipage subdomain. That check does not read robots.txt first.
Allow or block
it’s your call
Allow ChetverBot
Add this group to robots.txt:
User-agent: ChetverBot
Allow: /Block ChetverBot
Block the whole website:
User-agent: ChetverBot
Disallow: /Or block selected sections:
User-agent: ChetverBot
Disallow: /checkout/
Disallow: /account/Chetver cannot check the pages you block and marks those checks as unavailable.
Firewall, Cloudflare and rate limits
If your hosting firewall or security plugin blocks ChetverBot, review the blocked request and allow access only to the public pages you want checked.
In Cloudflare, a custom Skip rule can exempt requests from Super Bot Fight Mode. Scope it to public GET requests with the ChetverBot User-Agent, and skip only Super Bot Fight Mode. Keep managed rules, rate limiting and other security rules enabled.
If rate limiting blocks the check, review the threshold or narrow the rule to the intended public paths. Keep login, checkout, account and form pages protected for every User-Agent.
Cloudflare Bot Fight Mode cannot be skipped by a custom rule. If it blocks ChetverBot, review whether to disable that feature or use a bot protection mode that supports scoped exceptions.
Cloudflare: Super Bot Fight Mode exceptions · Bot Fight Mode
Questions about ChetverBot?
write to us
Send your website address and a description of the issue to support@chetver.com