It seems like the AI crawlers learned how to solve the Anubis challenges. Anubis is a tool hosted on our infrastructure that requires browsers to do some heavy computation before accessing Codeberg again. It really saved us tons of nerves over the past months, because it saved us from manually maintaining blocklists to having a working detection for "real browsers" and "AI crawlers".
Stefan Eissing
Lock
@icing@chaos.social
curl and Apache httpd project member, HTTP/2, HTTP/3, Lets Encrypt implementations. Likes to code.
Not toot authentic without spelling mistakes.
April 19, 2017
Show Full thread