Cloudflare makes a large step towards data monetization for AI training. Now all new data hosted on Cloudflare is inaccessible to AI crawlers by default. There is also a new option for the page to return code HTTP 402 (“Payment required”) and charge for access. Of course, Cloudflare will be an intermediary, which gives it significant control and financial power.
Moreover, it could spark an era of “dark crawling”, which will undoubtedly lead to cat-and-mouse games of crawer detection.