Cloudflare announced that they’re going to allow site owners to charge per-crawl fees. This is, one: shocking, because it flips the script for almost anyone doing any kind of crawling. Including search engines. And two: A power play given how large and ubiquitous Cloudflare is on the Internet. I’ll be interested in seeing how this actually plays out. It seems like a fair trade but one that may not be favored by crawlers, who will do anything to weasel out of this schema.

For now, Anubis is a real, working tool that you can deploy that (currently) just does Proof of Work checks against user-agents. I wonder if it could also implement this as a mechanism of bypassing the check for PoW if the agent paid.

Verrrry interesting.