OHW is under attack

[leggysman]

A distributed network of scraper bots (thanks AI guys!) is currently hammering OHW into the ground, and our server just can't cope.

I've tried all of my usual tricks, but blocking them is impossible. They operate incognito, and they're coming from all over the world - particularly South America and the Middle East - as many as 8000 of them at any given moment.

Presumably someone has introduced a scraper trojan into a mobile app, so the phones' owners are unaware that they're participating in this. I've seen the same thing happen to another forum I administer.

I've asked the management to consider sheltering OHW behind Cloudflare going forward, because this is a fight we can't win on our own.

In the meantime, sorry for all the aggravation. Know that we're aware of it, and trying to find a way to cope.

[johnswan]

The forum has been slow over the last few days, with occasional unwanted sign outs, etc.
Your description of what's happening probably explains those problems.

Presumably there's a monetary cost to installing and running Cloudflare?
If so, how would that be financed?

If Cloudflare is a solution, then the slight inconvenience of having that pop up during the sign in process would be worth it.

Hope you can quickly come up with a solution.

[leggysman]

The bots come and go in waves - for example after ~8h of the site being basically unusable, they backed off starting around 1am GMT. But they will most certainly be back.

It's not really a question of money; as I understand it, Cloudflare has a free tier for small sites like ours. And we have the technical know-how.

The primary problem is that the man (allengt) who owns the accounts at the domain registrar, and at the hosting provider, has seemingly passed away. The site being largely anonymous makes it difficult to prove that, transfer ownership, etc. armyguyot1 can pay the bills and whatnot, but changing anything will be difficult. The best long-term solution might be to let the domain registration expire, and then reacquire it.

Other solutions might include blocking whole countries, or disabling guest access to make all of the forums available to registered users only - both of which are extremely blunt tools - or me coding something up. The bots never request any images included in the pages; it seems they're only interested in the text. I might be able to use that against them.

[BigJim]

Thank you Leggysman for all your efforts and dedication to keep this great site alive.

[Bent_n_Twisted]

A distributed network of scraper bots (thanks AI guys!) is currently hammering OHW into the ground, and our server just can't cope.

I've tried all of my usual tricks, but blocking them is impossible. They operate incognito, and they're coming from all over the world - particularly South America and the Middle East - as many as 8000 of them at any given moment.

Presumably someone has introduced a scraper trojan into a mobile app, so the phones' owners are unaware that they're participating in this. I've seen the same thing happen to another forum I administer.

I've asked the management to consider sheltering OHW behind Cloudflare going forward, because this is a fight we can't win on our own.

In the meantime, sorry for all the aggravation. Know that we're aware of it, and trying to find a way to cope.

I don't know that cloudflare would be of any use. I am an admin for a kink-oriented site, we have been using cloudflare for several years now, I can't even get in. The issue started about the same time as the issue here, I'm guessing for the same reason. I can't even get to it, I just get a 'bad gateway' error (502) from cloudflare.

I checked some 'normal' sites but didn't seem to have any issues, just here and mine. Ping and traceroute 'seem' OK but responses appear to be in IPV6 format rather than dotted quad or resolved names (probably a cloudflare thing?).

We also had an ownership issue that somehow got straightened out, but I think the guy who has the keys to the kingdom is in Europe somewhere at the moment. I can't get ahold of him or the other admin. We should have been smart enough to ensure we had communication outside of the site, but, apparently not. DOH!

[leggysman]

Cloudflare will only be a help if it actually blocks the requests of these bots! There'll always be a cat-and-mouse game between the goodies and the baddies. In general though, I think it would ultimately help. If we can find a way to change the DNS entries for OHW.

Is your other site also phpBB? The gateway errors are phpBB's way of saying it didn't get a response from the database server. Which is exactly what happens when the database daemon is a) sick; or b) maxed-out, saturated with requests. I have root access to the OHW server using ssh, and I can buy a half a minute of good service by restarting these services, but that's not much use, really.

I haven't tried Cloudflare yet, but presumably ping and traceroute will only be informative if you're using the real IP address of your site. But, as I say, I think if you get in and have a look at the processes, you'll find that the database server is struggling.

What I'm thinking is: I could write a program that follows the web server log, and if a client has requested pages, but hasn't requested any image files, add it to a real time deny list. Remove a client from that list if it requests an image, and clear the list and start again when the logs rotate. It would be Nginx-specific (for our needs) but could probably be adapted for other http servers.

[leggysman]

Thank you Leggysman for all your efforts and dedication to keep this great site alive.

Cheers BigJim. I'm really glad to have been given the opportunity to help. This site means a lot to so many of us.

[BigJim]

Thank you Leggysman for all your efforts and dedication to keep this great site alive.

Cheers BigJim. I'm really glad to have been given the opportunity to help. This site means a lot to so many of us.

I agree

[Bent_n_Twisted]

Is your other site also phpBB? The gateway errors are phpBB's way of saying it didn't get a response from the database server. Which is exactly what happens when the database daemon is a) sick; or b) maxed-out, saturated with requests.

Yep, it is. We have been discussing moving to a different platform, but I don't think it was planned to happen until after the other guy gets back from Europe.

I've been thinking of trying to use some of the tools in Kali Linux but (a) I'm not real familiar with them at the moment and (b) with Cloudflare in the mix I'm not sure if there would be any legal liability in the attempt. I'm just at the beginning of trying to figure out what to do and I stopped keeping up on these things back in the late 90s/early 2000s so my knowledge is somewhat lacking at the moment...

[leggysman]

I've written and deployed an experimental program which attempts to do real-time blocking of bots, based on this principle that the bots really only request the PHP pages, and skip all the supplemental stuff like CSS files and images.

I'm not sure how effective it will be, but I guess we'll see when tonight's onslaught arrives.

There could be some collateral damage of innocent users getting blocked if their browsers are caching all of that supplemental stuff, but the blocking page has my email address on it, so hopefully they'll reach out if it happens. The program has a 'permit' list I can update by hand.

[leggysman]

I suppose a few innocents being blocked is preferable to the whole site becoming unusable, and having to take it offline again

Following this link and doing a page reload ought to unblock anybody after a few seconds, I think
https://ourhotwives.org/yhw_logo.gif

[leggysman]

I'm not sure how effective it will be, but I guess we'll see when tonight's onslaught arrives.

Answer: not very effective. My program had 1100 bots blocked, but by the time there were 8000 of them hitting the server at once, the site became unusable, and I took it offline.

[ but not before hiding 20 invisible megabytes of random text in the page we were serving up, just to eat up their bandwidth and hopefully deter them ]

I've tightened up my program's requirements, and already today it has 2200 bot IPs blocked, so we'll see if that's any better.

I've included an image in the blocked IP page, and text suggesting to first try reloading the page. Your browser downloading that image causes my program to auto-unblock your IP, and that works for me - my Tor is always changing my address and I frequently trigger the blocking mechanism - slightly irritating, but it's good to see it working as intended.

Next up might be to consider installing a 'tarpit' for these bots; code that sucks them in using a 'trap' URL and slowly serves them plausibly nonsensical text, and links to yet more static (but auto-generated) nonsense.

[ferrisandrews]

Ugh, sorry you have to deal with this. I ran my own ai report and refined a little based on my own previous related work and sent it in PM. I hope something can finally shut this nonsense out of the site!

[leggysman]

I ran my own ai report and refined a little based on my own previous related work and sent it in PM

Great stuff, thank you

[leggysman]

Still struggling, but still fighting.

To everyone context for this: in the past 12 hours, the site has been visited by more than 465K unique devices (different IP addresses).

Pretty sure we've never faced anything like this since I've been around.

One might surmise that a very popular phone app is secretly doing this in the background.

On an unrelated note, we had to uninstall the FB app from leggysandy's phone, because it was using up her monthly mobile data allowance in like 2 days, even though she said she wasn't using it much.

[Long Lurker 34]

Still struggling, but still fighting.

To everyone context for this: in the past 12 hours, the site has been visited by more than 465K unique devices (different IP addresses).

Pretty sure we've never faced anything like this since I've been around.

One might surmise that a very popular phone app is secretly doing this in the background.

On an unrelated note, we had to uninstall the FB app from leggysandy's phone, because it was using up her monthly mobile data allowance in like 2 days, even though she said she wasn't using it much.

LM
- I might be just a lurker but i certainly appreciate your hard work here trying to keep the ship righted.

[Greg_N_Shelley]

I propose moving everything behind the login wall until a more robust solution can be put in place, and putting a message on the welcome screen explaining the reason for the change. If I understand correctly, the bots will swarm and able to access a handful of pages. The attack is then over and the server overwhelm is over.

It may deter new members and interfere with Google indexing, but will preserve the community until a long term solution is achieved. My concern is the community will erode from attrition soon with the length and frequency of the locust swarms. And from what I gather, this isn’t going to end any time soon.

Leggysman deserves a medal!

[johnswan]

Agree with what Greg_N_Shelley said,
assuming that would help.

[craven mohr]

I will join the others in thanking you for your time and dedication on all things OHW, Leggysman!

There used to be something called "Shovel". It was a perl script running on a server. It was permissible to link it in my HTML / PHP. When a bot would scrape it, the triggered flow of data was *unending*. It was full of extended ASCII and plenty of "@" symbols so as to appear to be email addresses. It was a MESS.

I chuckled as I buried in a hidden link on my front page. Called it "email.htm"

Maybe this would help stuff them full~

[leggysman]

Thank you for all the support, everyone.

I don't think the management are too keen about the idea of moving everything behind the login, except possibly as a last resort. I've done that with the other forum I administer. The bots still show up in large numbers, but yeah - they're just churning through a few pages (FAQ, contact us) on an otherwise empty forum.

This Shovel sounds very much like a couple of 'tarpit' options, except the newer things can generate text that's designed to accelerate AI model collapse, because, well, fuck these guys. There are other, modern tarpits like Nepenthes and Sarracenia (thanks for the tip, Greg_N_Shelley)

My own program currently has 212K bots blocked for today. It does seem like it might be helping a bit.

A good solution proposed by ferrisandrews uses cached pages for guest (not logged in) users, so they're not hitting the database so hard. I implemented it last night, and it seemed like it suddenly, completely solved the problem. However, things got worse again later on. I might not have done it correctly though; our webserver configuration is slightly byzantine, so I'm going to have to study it to make sure the caching is doing what it's supposed to.

There are some other things to try, also. We'll solve it. Hopefully sooner rather than later - this is a drag.

[mrs_hotwifecplsa]

Thanks for your work!

When does the domain expire?

Making the site accessible only to registered users sounds like a good idea to me.

[leggysman]

When does the domain expire?

Later in the year (I'd rather not advertise that, even though anyone who's interested can find out).

My bot blocker seems to be relatively effective now, but I've given it another tweak; it was clearing the slate at midnight GMT when the logs rotate, but that's typically when the nightly bot surge is at full throttle. Now it won't do that, and I'll restart it manually when things are quiet. There are hundreds of thousands of bots who only hit the site a very few times each, so having a big list of them already in place is essential when they're swarming.

I've also refined the search tree so the bot blocker doesn't add so much of its own load to the server - I originally underestimated the size of the data set it would have to manage.

[leggysman]

it was clearing the slate at midnight GMT when the logs rotate

I think this is evidence that it was working decently well - because when the blocker would clear its list, the server soon became overloaded.

We'll see whether it's better tonight, although I won't be at my workstation to watch it at that precise moment

[craven mohr]

The site seems verrry much improved in the last 2 days or so. I think your fixes are quite effective.

Thanks again for all you do!~

[Late]

leggysman:
Thanks. It's all greek to me. All I know is I don't want OHW to go away. I guess maybe this will just make us appreciate the site more now.

Late