An analysis published today by Fastly suggests the percentage of web traffic being generated by artificial intelligence (AI) bots is growing at an alarming rate.

Based on analysis of more than 130,000 applications and application programming interfaces (APIs) generating more than 6.5 trillion requests per month, the report finds AI bots can, at peak traffic levels, drive up to 39,000 requests per minute.

AI crawlers, which operate similarly to search engine crawlers, accounted for 80% of all AI bot traffic, with roughly half (52%) of that attributed to Meta. AI fetchers that actually access website content in response to user actions, in contrast, only accounted for 20% of the AI bot traffic.

Additionally, the report notes ChatGPT generates the most real-time fetcher traffic to websites, with 98% of fetcher bot requests attributable to OpenAI.

Arun Kumar, a senior security researcher at Fastly, said that while AI bots still generate a comparatively small amount of all bot traffic generated, they are having an impact on the cost of owning a website. In fact, there have already been reports of websites crashing because they were overwhelmed by AI bot traffic, he added. In one instance, the Fastly report notes a single crawler reached a peak of around 1,000 requests per minute. In another instance, a fetcher bot made 39,000 requests per minute to a single website at peak load.

Most AI bot traffic is scraping content from websites in the commerce, media and entertainment, and high tech sectors, according to Fastly. It’s unclear what the total cost of processing all that additional traffic is, but the need for bot management capabilities is becoming more acute, said Kumar.

About 87% of bot traffic is generally malicious, but now IT teams will need to monitor a significant amount of additional bot traffic that is not necessarily providing meaningful business value, he added.

Each organization will need to determine to what degree to allow AI bots to access their websites. AI fetchers, for example, are typically performing a task that has been assigned via a specific end-user request. AI crawlers, in contrast, are typically looking for more data to train an AI model. A fine balance may need to be struck between wanting AI models to be aware of a website and how much it costs to serve up data to AI crawlers.

Cybersecurity teams should also be analyzing that traffic because cybercriminals are also becoming more adept at setting up fake AI bots that might be used to collect data or otherwise evade security controls for a wide range of malicious purposes, noted Kumar.

It’s not clear to what degree the rise of AI bot traffic might drive more organizations to rely on content delivery networks (CDNs) to manage web traffic but as AI platforms other than ChatGPT start to gain additional traction the volume of AI bot traffic is only going to continue to exponentially increase to the point where processing it becomes a significant economic burden for organizations. As it is unlikely the organizations that have created those AI bots are going to help defer those costs, it will be up to each organization to determine how much of a privilege it might be for them to help train AI models that continue to increase in terms of not only number but also their appetite for data.