Eight in ten of world’s biggest news websites now block AI training bots
Picture: Shutterstock/Ketmut Almost eight in ten of the biggest news websites in the UK and US are now blocking AI training bots. This week in London publishers were told it is “never too late” to begin blocking the bots due to the frequency with which new crawling needs to take place via retrieval-augmented generation so that AI chatbot responses remain current. Some 79% of almost 100 top news websites in the UK and US are blocking at least one crawler used for AI training out of OpenAI’s GPTBot, ClaudeBot, Anthropic-ai, CCBot, Applebot-Extended and Google-Extended. Meanwhile 71% are blocking AI bots from crawling their sites for retrieval or live searches. Those bots are: ChatGPT-User, Claude-Web, Perplexity-User and OAI-SearchBot. The latest analysis of who was blocking what was carried out by digital PR platform Buzzstream, looking at a combined and deduplicated list of the 50 biggest news websites in each of the UK and US, and shared with Press Gazette. Among the 50 biggest, the publishers that had blocked all the AI bots included in the analysis …






