☆ Yσɠƚԋσʂ ☆@lemmy.ml to Open Source@lemmy.mlEnglish · 5 days agoOpen Source Infrastructure is Breaking Down Due to Corporate Freeloadingnews.itsfoss.comexternal-linkmessage-square20linkfedilinkarrow-up1160arrow-down12
arrow-up1158arrow-down1external-linkOpen Source Infrastructure is Breaking Down Due to Corporate Freeloadingnews.itsfoss.com☆ Yσɠƚԋσʂ ☆@lemmy.ml to Open Source@lemmy.mlEnglish · 5 days agomessage-square20linkfedilink
minus-squarerenegadespork@lemmy.jelliefrontier.netlinkfedilinkEnglisharrow-up73·5 days agoThere’s no excuse for this crap. Even if they insist on scraping every FOSS repo, there needs to be some logic to it (caches, diffs, longer intervals). These AI scrapers are so poorly thought out they are indistinguishable from DOS attacks.
minus-squareDigitalDilemma@lemmy.mllinkfedilinkEnglisharrow-up29·5 days agoYou’re not wrong. Claudebot took down one of my sites repeatedly, hammering it for the same pages over and over at horrendous rates. I ended up spending several days having to convert it to a SSG and hosted it on Cloudflare pages. A lot of work I didn’t need to do.
There’s no excuse for this crap. Even if they insist on scraping every FOSS repo, there needs to be some logic to it (caches, diffs, longer intervals). These AI scrapers are so poorly thought out they are indistinguishable from DOS attacks.
You’re not wrong.
Claudebot took down one of my sites repeatedly, hammering it for the same pages over and over at horrendous rates.
I ended up spending several days having to convert it to a SSG and hosted it on Cloudflare pages. A lot of work I didn’t need to do.