While adhering to the standard isn’t required by law, most internet firms follow the protocol. Compliance is also mandatory for websites that rely on Amazon Web Services, such as Perplexity.
An Amazon Web Services spokesperson said, “AWS’s terms of service prohibit abusive and illegal activities, and our customers are responsible for complying with those terms.
We routinely receive reports of alleged abuse from a variety of sources and engage our customers to understand those repor
Scrutiny of Perplexity’s practices has intensified after Forbes accused the company earlier this month of “directly ripping off” articles written by its reporters and others by CNBC and Bloomberg, including those behind paywalls.
Wired approached Amazon after its investigation, which determined that Perplexity allegedly used an “unpublished IP address” to scrape websites operated by its parent company, Condé Nast — even though it was trying to block access.
Comment
byu/Maxie445 from discussion
intechnology
The outlet said representatives from other outlets, including Forbes, the New York Times and the Guardian, had detected the same IP address visiting their servers.
Perplexity spokesperson Sara Platnick pushed back on Wired’s report, calling it “inaccurate.”
“Our PerplexityBot — which runs on AWS — respects robots.txt, and we confirmed that Perplexity-controlled services are not crawling in any way that violates AWS Terms of Service,” Platnick said in a statement.
“AWS looked into WIRED’s media query as part of a standard protocol for investigating reports of abuse of AWS resources,” Platnick added. “We had not heard anything from AWS prior to a WIRED reporter contacting them. To say that AWS is ‘investigating’ Perplexity outside of this specific WIRED inquiry is incorrect. AWS is a valuable partner to Perplexity, and we are grateful for their ongoing collaboration.”
Platnick told Wired that the PerplexityBot would bypass the robots.txt protocol in “very infrequent” circumstances where a user included a specific URL in their query.
Perplexity CEO Aravind Srinivas had previously slammed Wired’s findings, asserting that they “reflect a deep and fundamental misunderstanding of how Perplexity and the Internet work.”
Comment
byu/Maxie445 from discussion
intechnology
Forbes had taken issue with a feature called “Perplexity Pages,” a product that displays “curated” articles that pull details from articles written by third-party news outlets.
The original authors weren’t credited by name, even when the wording of Perplexity’s posts closely matched that of the source text.
Comment
byu/TheFourthIteration from discussion
inperplexity_ai
Instead, Perplexity used what Forbes described as “small, easy-to-miss logos”, linking back to the original sources. Perplexity’s chatbot churned out a version of an exclusive, paywalled Forbes report on ex-Google CEO Eric Schmidt’s military drone project in one egregious example.
“Our reporting on Eric Schmidt’s stealth drone project was posted this AM by @perplexity_ai,” Forbes Executive Editor John Paczkowski wrote on X at the time. “It rips off most of our reporting. It cites us, and a few that reblogged us, as sources in the most easily ignored way possible.” Srinivas said the tool “has rough edges” but otherwise denied wrongdoing.
In response to these allegations, Perplexity AI has argued that incidents of content misrepresentation might result from the nature of the prompts given to the AI and not deliberate oversight.
Srinivas emphasized that the Robots Exclusion Protocol is not legally binding and suggested the need for a new framework to govern AI interactions with web content.
Comment
byu/TheFourthIteration from discussion
inperplexity_ai
This case underscores broader challenges in the AI industry, where other companies like OpenAI and Anthropic have also faced criticism for allegedly ignoring robots.txt signals.
The investigation by Amazon and potential legal actions from publishers like Forbes may lead to new norms and regulations governing AI interactions with copyrighted material, ensuring a fairer ecosystem for content creators and AI developers alike.
For more news and insights, visit AI News on our website.
