Future Retirement Success
  • Politics
  • Business
  • Investing
  • Stocks
  • Politics
  • Business
  • Investing
  • Stocks

Future Retirement Success

Business

Nearly a third of the world’s biggest news sites have blocked AI crawlers from accessing their content

by September 19, 2023
September 19, 2023
Nearly a third of the world’s biggest news sites have blocked AI crawlers from accessing their content

After the ChatGPT launch in November last year, companies and consumers worldwide started using generative artificial intelligence (AI) to automate tasks, write documents, do market research, or even basic coding.

However, the rise of large language models and generative AI has also pushed into the spotlight the problem of news sites, publishers, and intellectual property holders who see their data being collected by AI crawlers. And while there are still no clear regulatory rules controlling AI’s use of copyrighted material, some of the world`s largest news websites have taken matters into their own hands.

According to data presented by AltIndex.com, nearly one-third of the world’s top 50 news sites have blocked AI crawlers from accessing their content, and their number continues rising.

CNN, New York Times, Daily Mail, Reuters, and Bloomberg have all Blocked at least one AI crawler

AI companies send crawlers to collect data to train their models and provide information for chatbots. However, as data is one of their core advantages, many of the world’s largest news websites have become extremely cautious, especially since there is generally no upside to handing over their data to AI crawlers.

The entire situation escalated last month after OpenAI had launched its GPTBot crawler to collect data to enhance its language models. Although the AI company promised that paywalled content would be excluded from websites, several high-profile news sites, including CNN, Reuters, and the New York Times, blocked GPTBot. Their number continued growing in the following weeks.

According to a Kirwan Digital Marketing Agency survey, 28% of the top 50 news sites worldwide have blocked at least one AI crawler by the end of last month. In regional comparison, the picture is a bit different. For example, 24%, or twelve out of fifty largest news sites in the United States, have blocked at least one AI crawler, far more than in the United Kingdom, where only three of 21 leading sites did the same. In India, the percentage of top new sites unwilling to hand over their data to AI companies is much higher, with one-third blocking at least one AI crawler.

One in Five Top News Sites has Blocked GPTBot

Although most of the world’s 50 leading news sites still haven’t taken action on blocking, the study showed GPTBot is the number one choice among those who have. Statistics show the brainchild of OpenAI has been blocked 22% of the time across the top 50 news sites, with Bloomberg, Reuters, Business Insider, Washington Post, the New York Times, and CNN as the top names on this list.

CCBot has been blocked about half as often as the GPTBot, with a 10% share across the top 50 news sites. The survey also showed ChatGPT had been blocked by only one website, that of the Washington Post, the same as AnthropicAI, which has been blocked by only the UK’s NewsNow.

Overall, the New York Times, Washington Post, Reuters, and NewsNow lead in blocking AI crawlers from accessing their content, with each news site blocking two AI bots.

Read more:
Nearly a third of the world’s biggest news sites have blocked AI crawlers from accessing their content

0
FacebookTwitterGoogle +Pinterest
previous post
Susan Collins jokes she will ‘wear a bikini’ to the Senate while slamming dress code changes
next post
Young entrepreneur, George Jones, transforms £11m Manchester office block into thriving business hub

You may also like

UK takeover surge hits £74bn in first half...

July 2, 2025

Jeremy Hunt ‘could have an extra £11bn for...

December 12, 2023

Christmas party cancellations ‘near Omicron level amid UK...

December 9, 2022

UK’s budding entrepreneurs stifled by cost of living...

December 21, 2023

Business Champion Award’s release SME of the Year...

January 30, 2023

Nadhim Zahawi sacked as Tory chairman by Rishi...

January 29, 2023

Southeastern trains subsidy rises to £415m, more than...

October 7, 2024

Taxpayers lose largest-shareholder status in Natwest as Blackrock...

March 1, 2025

Barclays launches £22bn fund and new business prosperity...

November 13, 2024

Raspberry Pi Shares Surge on London Debut, Sparking...

June 12, 2024

    Get free access to all of the retirement secrets and income strategies from our experts! or Join The Exclusive Subscription Today And Get the Premium Articles Acess for Free

    By opting in you agree to receive emails from us and our affiliates. Your information is secure and your privacy is protected.

    Recent Posts

    • National Wealth Fund commits £200m to UK battery storage push

      August 28, 2025
    • XRP price prediction hits new highs, GMO Miner helps you earn $6,800 a day

      August 28, 2025
    • West End retailers lose £310m from VAT-free shopping ban in first half of year

      August 28, 2025
    • Dollar slips as Trump moves to sack Fed governor Lisa Cook in unprecedented clash over central bank independence

      August 28, 2025
    • CDC Director Susan Monarez refuses to be fired as other officials call it quits

      August 28, 2025
    • Bill Gates met with Trump to talk ‘importance of US global health programs and health research’: spox

      August 28, 2025

    Categories

    • Business (8,901)
    • Investing (2,246)
    • Politics (16,503)
    • Stocks (3,228)
    • About us
    • Privacy Policy
    • Terms & Conditions

    Disclaimer: futureretirementsuccess.com, its managers, its employees, and assigns (collectively “The Company”) do not make any guarantee or warranty about what is advertised above. Information provided by this website is for research purposes only and should not be considered as personalized financial advice. The Company is not affiliated with, nor does it receive compensation from, any specific security. The Company is not registered or licensed by any governing body in any jurisdiction to give investing advice or provide investment recommendation. Any investments recommended here should be taken into consideration only after consulting with your investment advisor and after reviewing the prospectus or financial statements of the company.

    Copyright © 2025 futureretirementsuccess.com | All Rights Reserved