Your site. Your crawl settings.
PurityBot reads public pages to build Purili's independent search index. You can stop crawling with robots.txt, or ask us to add your domain to our opt-out or direct-only list.
Opt out with robots.txt
Add these lines to https://example.com/robots.txt at your website's root. Keep rules for other crawlers intact:
User-agent: PurityBot Disallow: /
Serve the file directly as plain text with HTTP 200. Apply it to every hostname and protocol you use, including www and other subdomains. We refresh cached robots rules within 24 hours. A manually confirmed opt-out stops new requests once our list refreshes, normally within five seconds; requests already in flight may finish.
To block only a section, use Disallow: /private/. To allow a section inside it, add a more specific Allow: /private/public/. An empty Disallow: does not block crawling.
Robots.txt is public guidance, not access control. Protect private content with authentication. If robots.txt is unreachable, returns a server error, or redirects, we defer crawling; a missing file (404 or 410) provides no crawl restrictions.
Request a manual setting
Email hello@puri.li with your domain and either “stop crawling” or “direct-only”. We verify ownership, for example with a verification file on your website, and confirm when the setting is active. No account or DNS record is required.
A listed domain includes its subdomains. Tell us if you control only a subdomain; we restrict changes to the scope you can verify. You can use robots.txt yourself without waiting for manual review.
Check your crawl settings
We maintain opt-outs and direct-only requests manually. Email hello@puri.li with your domain and preferred setting. We confirm site ownership before changing the list.
For direct-only crawling, allow 138.201.57.92/32 in your firewall or web server for HTTP/HTTPS. Cloudflare is not required. Test access below and wait for our confirmation before relying on direct-only mode.
Checking does not change your setting. The access test requests robots.txt and your chosen page from our IPv4 address, respects opt-outs and does not index the page. IPv6-only sites cannot use this mode.
Verify that a request is ours
Our crawler identifies itself with:
PurityBot/1.0 (+https://puri.li/bot)
A User-Agent can be copied by anyone. For a verifiable source address, request direct-only crawling and wait for our confirmation and compare the actual network peer IP with 138.201.57.92. That mode binds both page and robots.txt requests to this IPv4 address and never retries through Cloudflare Workers, including after redirects.
Use your existing firewall, Nginx, Apache, or hosting access controls; Cloudflare is not required. If you use a reverse proxy, use only the client address supplied by your trusted proxy configuration. Do not trust arbitrary X-Forwarded-For headers.
In normal mode, requests may also come from Cloudflare Workers. Shared Cloudflare IPs and the User-Agent alone do not prove a request is PurityBot.
The machine-readable direct-mode list is at /bot/ips.json. The current allowlist is:
138.201.57.92/32
What happens to existing search results?
Stopping crawling prevents future fetching; it does not automatically remove pages already in the index. The access test also does not submit a page for indexing.
For removal of existing results, or if you cannot edit robots.txt, email hello@puri.li with the exact domain or URLs. We may request proof of site ownership before removing results, so a third party cannot remove your site.
To resume normal crawling, email us to remove your manual restriction and remove any blocking robots.txt rules. Direct-only mode never overrides robots.txt.