Perplexity revealed a response to Cloudflare’s claims that it disrespects robots.txt and engages in stealth crawling. Perplexity argues that Cloudflare is mischaracterizing AI Assistants as net crawlers, saying that they shouldn’t be topic to the identical restrictions since they’re user-initiated assistants.
Perplexity AI Assistants Fetch On Demand
In response to Perplexity, its system doesn’t retailer or index content material forward of time. As an alternative, it fetches webpages solely in response to particular consumer questions. For instance, when a consumer asks for current restaurant critiques, the assistant retrieves and summarizes related content material on demand. This, the corporate says, contrasts with how conventional crawlers function, systematically indexing huge parts of the online with out regard to fast consumer intent.
Perplexity in contrast this on-demand fetching to Google’s user-triggered fetches. Though that’s not an apples-to-apples comparability as a result of Google’s user-triggered fetches are within the service of studying textual content aloud or website verification, it’s nonetheless an instance of user-triggered fetching that bypasses robots.txt restrictions.
In the identical manner, Perplexity argues that its AI operates as an extension of a consumer’s request, not as an autonomous bot crawling indiscriminately. The corporate states that it doesn’t retain or use the fetched content material for coaching its fashions.
Criticizes Cloudflare’s Infrastructure
Perplexity additionally criticized Cloudflare’s infrastructure for failing to differentiate between malicious scraping and legit, user-initiated visitors, suggesting that Cloudflare’s method to bot administration dangers overblocking providers which can be appearing responsibly. Perplexity argues {that a} platform’s incapability to distinguish between useful AI assistants and dangerous bots causes misclassification of official net visitors.
Perplexity makes a robust case for the declare that Cloudflare is obstructing official bot visitors and says that Cloudflare’s choice to dam its visitors was based mostly on a misunderstanding of how its know-how works.
Learn Perplexity’s response:
