Perceptful.AI's fetchers

What is this?

Perceptful.AI retrieves individual web pages, PDFs, and RSS/Atom feeds on behalf of our users, under one of two identities, depending on whether a person is waiting on the result of that specific request:

• PerceptfulAI-Bot - scheduled or recurring fetches where no one is watching in real time, such as polling an RSS or Atom feed you publish.
• PerceptfulAI-User - a fetch made because a specific person is waiting on this exact request right now, such as adding a URL to their library or asking our AI research assistant a question.

Neither is a general-purpose web crawler. Neither discovers or traverses links on its own, beyond following at most one discovered feed link when resolving a site URL to its feed, and both only ever fetch URLs a user has supplied, subscribed to, or that our assistant selected from search results while answering that user's question.

Why did I see a request from PerceptfulAI-Bot?

A user subscribed to one of your RSS or Atom feeds, and this fetch is a routine poll for new articles.

Why did I see a request from PerceptfulAI-User?

Typically one of the following happened:

• A user added one of your pages or PDFs to their document library.
• A user asked our AI research assistant a question, and it followed a link — possibly one it selected from search results — to read the page while preparing an answer.
• A user is viewing, in their browser, a document we fetched on their behalf.

How to identify them

Every request identifies itself with one of these exact User-Agent strings:

PerceptfulAI-Bot/1.0 (+https://perceptful.ai/fetcher)
PerceptfulAI-User/1.0 (+https://perceptful.ai/fetcher)

We do not spoof browser User-Agent strings or otherwise disguise these requests as human traffic.

Behavior

• Fetches a single URL per request — no crawling or link traversal beyond the single feed-discovery hop described above.
• Does not execute JavaScript or render pages; it reads the raw HTTP response.
• PerceptfulAI-Bot's feed polling uses conditional GET (If-None-Match / If-Modified-Since) so unchanged feeds don't re-transfer their full body.
• Follows a bounded number of redirects and applies a reasonable per-request timeout and response-size cap.

robots.txt

PerceptfulAI-Bot honors your robots.txt — disallow rules will stop its scheduled feed polling. PerceptfulAI-User does not check robots.txt, since each request is relaying a single fetch a specific person is waiting on right now, the same way a person clicking a link in their own browser would not be.

Blocking these fetchers

If you'd prefer PerceptfulAI-Bot or PerceptfulAI-User not access your site, please contact us with the domain and we'll stop fetching it.