Got a warning for my blog going over 100GB in bandwidth this month… which sounded incredibly unusual. My blog is text and a couple images and I haven’t posted anything to it in ages… like how would that even be possible?

Turns out it’s possible when you have crawlers going apeshit on your server. Am I even reading this right? 12,181 with 181 zeros at the end for ‘Unknown robot’? This is actually bonkers.

Edit: As Thunraz points out below, there’s a footnote that reads “Numbers after + are successful hits on ‘robots.txt’ files” and not scientific notation.

Edit 2: After doing more digging, the culprit is a post where I shared a few wallpapers for download. The bots have been downloading these wallpapers over and over, using 100GB of bandwidth usage in the first 12 days of November. That’s when my account was suspended for exceeding bandwidth (it’s an artificial limit I put on there awhile back and forgot about…) that’s also why the ‘last visit’ for all the bots is November 12th.

  • pendel@feddit.orgEnglish
    16·
    6 days ago

    I had to pull an all nighter to fix some unoptimized query because I had just launched a new website with barely any visitors and hadn’t implemented caching yet for something that I thought no one uses anyway, but a bot found it and broke my entire DB through hitting the endpoint again and again until nothing worked anymore

  • Omega_Jimes@lemmy.caEnglish
    3·
    5 days ago

    It’s a shame we don’t have those banner ad schemes anymore. Cybersquatting could be a viable income stream if you could convince the cleaners to click banner ads for a faction of a penny each.

  • ChaoticNeutralCzech@feddit.orgEnglish
    3·
    5 days ago

    I don’t know what “12,181+181” means (edit: thanks @Thunraz@feddit.org, see Edit 1) but absolutely not 1.2181 × 10185. That many requests can’t be made within the 39 × 109 bytes of bandwidth − in fact, they exceed the number of atoms on Earth times its age in microseconds (that’s close to 1070). Also, “0+57” in another row would be dubious exponential notation, the exponent should be 0 (or omitted) if the mantissa (and thus the value represented) is 0.

    • benagain@lemmy.mlOPEnglish
      1·
      3 days ago

      My little brain broke when I started trying to figure out how big the number was… thanks for breaking it down even more intuitively, yeah it is way to large to have been correct!

  • WolfLink@sh.itjust.worksEnglish
    91·
    7 days ago

    This is why I use CloudFlare. They block the worst and cache for me to reduce the load of the rest. It’s not 100% but it does help.

    • irmadlad@lemmy.worldEnglish
      4·
      7 days ago

      LOL Someone took exception to your use of Cloudflare. Hilarious. Anyways, yeah, what Cloudflare doesn’t get, pFsense does.

  • KeenFlame@feddit.nuEnglish
    5·
    6 days ago

    What is the blog about? It may be increased interest as search providers use them for normal searches now… or it could be a couple of already sentient doombots.

    Please don’t be a blog about von Neumann probes. Please don’t be a blog about von Neumann probes. Please don’t be a blog about von Neumann probes…

    • benagain@lemmy.mlOPEnglish
      4·
      7 days ago

      It’s a mix, I put two screenshots together. On the left is my monthly bandwidth usage from CPanel on the right is Awstats (though I hid some sections so the Robots/Spiders section was closer to the top).

        • benagain@lemmy.mlOPEnglish
          1·
          6 days ago

          I think they’re winding down the project unfortunately, so I might have to get with the times…

          • [object Object]@lemmy.worldEnglish
            1·
            6 days ago

            I mean, I thought it was long dead. It’s twenty-five years old, and the web has changed quite a bit in that time. No one uses Perl anymore, for starters. I used Open Web Analytics, Webalizer, or somesuch by 2008 or so. I remember Webalizer being snappy as heck.

            I tinkered with log analysis myself back then, peeping into the source of AWStats and others. Learned that a humongous regexp with like two hundred alternative matches for the user-agent string was way faster than trying to match them individually — which of course makes sense seeing as regexps work as state-machines in a sort of a very specialized VM. My first attempts, in comparison, were laughably naive and slow. Ah, what a time.

            Sure enough, working on a high-traffic site taught me that it’s way more efficient to prepare data for reading at the moment of change instead of when it’s being read — which translates to analyzing visits on the fly and writing to an optimized database like ElasticSearch.

  • ChaoticNeutralCzech@feddit.orgEnglish
    1·
    5 days ago

    That’s insane… Can’t a website owner require bots (at least those who are identifying themselves as such) to prove at least they’re affiliated with a certain domain?

  • hdsrob@lemmy.worldEnglish
    3·
    7 days ago

    Had the same thing happen on one of my servers. Got up one day a few weeks ago and the server was suspended (luckily the hosting provider unsuspended it for me quickly).

    It’s mostly business sites, but we do have an old personal blog on there with a lot of travel pictures on it, and 4 or 5 AI bots were just pounding it. Went from 300GB per month average to 5TB on August, and 10/11 TB in September and October.