Reading Time: 6 minutes
The age of bots is here. While we have always been awash with automated crawlers and scrapers, it has now become overwhelming. For a long time, the reporting of website statistics in annual reports or as a service benchmark for law libraries was questionable. Now it is downright pointless.
There was always a weakness in using website analytics as a proxy for value. I have worked in enough organizations who got this wrong—while usually working on the website team that would know better—to be confident in this. A report to a governance board that discusses “visitor increase of X” or “pageviews of Y” and comparing them to prior month or year-over-year was always missing the point.
Web analytics still had a purpose. They could indicate whether a resource was accessed at all. Or they might be used for campaign metrics, to see if someone was visiting your website based on a marketing event. While there was already a softness to the analytics, they still had some tactical value.
In commercial organizations, they would not (probably?) make that error. They would focus on conversions: visitors had pageviews and then bought a product or took an action. The sale or the action would be the measurement that mattered. To a certain extent, this commercial outcome inoculated some website properties from the misuse of web analytics.
To some extent, then, I was never that invested in search engine optimization or the vagaries of the Google search algorithm. Before the emergence of artificial intelligence in web search results, a change in the algorithm was usually uneventful for me. It caused some consternation for people who constantly re-tune their websites to the algorithm, who see a significant drop after a search engine change. This sort of SEO requires a lot of work, gaming the system to get your website to appear in the top results. I have seen steady visitors over the 30 or so years I’ve had this website but then again, I’m not trying to live off of it.
Animal, Vegetable, or Mineral
For a long time, like so many things on the internet, there was a community agreement about visitors. Website owners like me would publish a text file, called robots.txt (here’s mine) that indicated what we were comfortable having searched and what we weren’t. Search engines and other crawlers would, normally, access that file first and determine what was in or out of bounds. For years and years, mine was empty: any service could crawl my site, any part of it that they wanted.
Over time, I saw more and more crawlers for search engine optimizers. Services that were checking backlinks and other SEO minutiae. I didn’t really care but I took notice that this traffic was no longer a human. Automated tools were crawling the site. Again, not a surprise but now, where a search engine crawler might visit, I could no longer expect someone might benefit from that by seeing a search result. The crawling wasn’t contributing to people finding things. The trade-off was starting to get shaved thin.
Now, we are starting to see websites go offline because the crawlers and automated tools are being directed without any thought. The carbon-based website visitors are being overwhelmed by the silicon-based army. The animals are being replaced by the minerals. Scrapers like Perplexity have been found to be ignoring the robots.txt (they deny this) or, rather, not ignoring it so much as trying to subvert it. Brave’s search does the same thing: ignores robots.txt and uses what they call a “generic” user-agent so they cannot be blocked so as not to be “discriminated against”.
This leads to an ongoing adversarial approach. Now, anyone relying solely on robots.txt will find their website overrun by bots. What does that mean? At the basic level, it means that the servers will be requested for files, whether those files exist or not, and the servers will attempt to respond. Each response requires a bit of CPU, a bit of memory, and all of those resources add up.
It’s not just that they metaphorically add up. They have an actual cost to the website owner. When a request is made to the server, animal or mineral, it has a cost. A website owner will plan for that. At on point, website analytics could help with that: you could see the number of views and visitors—whether or not they converted—and could size your hardware to meet that need. You could have a decent guess about what it would cost you to host your website.
No longer.
This chart is a slightly compressed version of my Cloudflare dashboard security rules. You can see that there were 7,720 requests in a 24 hour period that were blocked outright. Another 5,720 in that same period were challenged, and failed. Those two spikes on the first rule? They were coming as often as every 10 seconds from all over the world.
This is my own website, where I used to get maybe a few hundred visits a day. Now, in addition to normal traffic, I’m blocking 10x that in automated requests. Legitimate website crawlers like the old Google or Bing had created new systems, using sitemaps, that meant that they no longer needed to crawl the site. When a new page was published, the sitemaps would update, Bing (through its Webmaster tools) and Google (through Search Console) would notice and would index that new page.
It is extraordinarily frustrating. I have posted before about how adversarial it has become. Since these automated scrapers are hosted by the same companies that host network security services, including Microsoft and Amazon, it means that blocking a scraper can also block a legitimate human visitor. I have been so grateful to those of you who notify me when you can no longer see a post. It has led me to regularly trying to balance “block v. challenge” to try to ensure as many humans can get through.
Think of the time that is being wasted. You may be frustrated as a visitor every time you open a new website and have to verify you are human. Imagine how much work has gone into trying to let you see even that. Services like Scraper.do and Scrapify are an example of the types of things website operators are combatting.
I’ve seen a mixture of these on my own website:
- headless browsers act as if they are a web browser but have no graphical interface. It is a program that sends a request without any visual interaction. There’s no person on the other end viewing the page.
- Proxy rotations will send repetitive requests from a variety of different locations. For awhile, I had a bunch of requests coming from Afrinic (an African registrar) IP addressess pretending to be from US locations (and who knows where the requester was). But you will usually see them in a cluster, the same page request but from four or five different IP addresses and locations. The scraper is making the same request over and over, rotating their fake location each time. You can see that below (all of those IP addresses have been added to my blocking list).
It used to be that a managed challenge would stop most of this, the captchas that people are completing. Captchas are no longer doing the work, except perhaps against people. This article discusses an AI bot that, in 2024, could solve image recaptchas with 100% accuracy. Cloudflare’s Turnstile product, which will be familiar to you if you’ve seen a “verify” checkbox instead of a “how many bicycles” photo, is targeted as well. Web application firewalls like Cloudflare and Amazon will provide some blocking but they are, for the most part, not offering out of the box configurations. A website owner will need to do that themselves.
All for the sake of a bunch of bros training data for AI.
Measurement Mirage
It’s incredible how quickly this has all fallen apart. Even this time last year, it hadn’t gotten so bad. For many yeas, I would keep track of website visits to a particular web page as a way to acknowledge interest in a topic. When the AALL KnowItAALL daily newsletter picked up a post, I could see that human traffic spike. Outside of that, though, I could see domains and visitors who were clearly coming for information and, I think, often finding it. The people who wanted to be build an Irish dance stage or solve a computer problem. Canadian lawyers who have been coming every few months for the last decade to learn how to make a simple tweak to improve the Canadian immigration form.
Increasingly, though, there is no way to look at a visitor or pageview count and have any idea if anyone actually saw a web page. A human, I mean. Those numbers are so grossly unreliable as to be useless now. Where an uninformed governance board might be overwhelmed with a 20% increase in web usage, year-over-year, anyone suggesting that’s happening now will need a lot of additional data to prove it. It’s one of the reasons that I found services like Elsevier’s BePress to be so preposterous in showing downloads. Any simple system that is looking solely at what and where and who for a website visitor is going to be sorely mistaken.
I’m disappointed. I don’t write this blog for views but it has been nice, over the years, to see people get some use out of it. That one person who seemed curious about the use of anvils in music or even that law librarian who told me they cited something on the website while in library school. Maybe they would even leave a comment. Now I’m not sure those people will ever find what’s on my site. Either their search will be overwhelmed with reconstituted SEO (AEO and GEO) or artificial intelligence, or, to protect my own systems and to be able to afford to have a website, I’ll end up blocking avenues they might have followed to get to it.
As someone who values analytics, it’s also frustrating. Web analytics were often misused but they were a tool for website planning. You could build your architecture with them in mind. Recently, someone told me that their corporate website is going down about 20 times a day because of bots. Now they are having to re-scale their hardware to meet that demand but I don’t think that’s going to work. If you try to adapt to absorb all the potential bots, your website may become so costly as to eliminate any return on the investment.
Website analytics are largely for me to see when some scraper has broken containment. It has become a joyless resource and I feel sorry for the next person who pitches me based on website views and visit data. It will be very hard for me to take them seriously.