[10:34:40] <TheresNoTime>	 hihi, is https://en.wikipedia.beta.wmcloud.org down for everyone or just me :)
[10:36:07] <TheresNoTime>	 (just saw the alerts firing in -releng, not just me)
[10:37:22] <taavi>	 TheresNoTime: broken for me as well
[16:19:53] <Southparkfan>	 Quick Q, any ETA on the approval of rejection for the quota request in T437273? Bullseye deprecation deployment-prep is blocked now, until the # of server groups is raised
[16:19:53] <stashbot>	 T437273: Regression: CentralAuth doesn't support Read-Only mode any more - https://phabricator.wikimedia.org/T437273
[16:20:35] <Southparkfan>	 Err, I meant T437117 ...
[16:20:36] <stashbot>	 T437117: Quota increase request for project deployment-prep - https://phabricator.wikimedia.org/T437117
[16:27:12] <dcaro>	 Southparkfan: it was approved, just waiting for the clinic duty to pick it up I guess, they should take tops a week (we discuss them in the weekly team meeting)
[17:34:48] <Southparkfan>	 dcaro: alright, thank you! Looking forward to the increase :-)
[18:33:20] <andrewbogott>	 !log dumps shutting down dumps-10 as there is no response on T435767
[18:33:24] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Dumps/SAL
[18:33:24] <stashbot>	 T435767: Migrate dumps away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435767
[18:49:07] <andrewbogott>	 !log admin restarting haproxy on proxy-5.project-proxy due to port exhausting.  This will likely happen again in a couple hours.
[18:49:15] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[19:03:51] <andrewbogott>	 musikanimal: are you involved in the wikisource project? It's getting a zillion connections right now and I'm trying to get some visibility into what's happening.
[19:05:12] <musikanimal>	 andrewbogott: not anymore. Which tool? If it's WS Export, I suspect it's a XTools-esque bot attack (this happens to all tools that are linked to from the wiki)
[19:05:35] <andrewbogott>	 yep, wsexport
[19:05:59] <andrewbogott>	 by 'xtools-esque' do you mean the attacker is xtools-esque, or that the attack is similar to attacks that xtools has faced?
[19:07:15] <lucaswerkmeister>	 ohhhh, is that why xtools gets scraped so much (I didn’t know about the “linked from the wiki” bit before, or rather didn’t make that connection)
[19:07:24] <musikanimal>	 the latter. I even pinged Samwilson in https://phabricator.wikimedia.org/T430876 as I thought the mitigation strategies we used there could apply to WS-Export, too
[19:09:04] <musikanimal>	 there's likely two things that would help immediately: (1) upgrade Anubis and (2) employ use of mod_proxy_cgi so that Apache doesn't get flooded with requests https://phabricator.wikimedia.org/T430888#12080489
[19:09:34] <musikanimal>	 this is a routine, "normal" problem for WS-Export https://stats.uptimerobot.com/BN16RUOP5/782558466
[19:09:40] <andrewbogott>	 does xtools use the standard cloud-vps proxy, or does it have a floating IP?
[19:09:56] <andrewbogott>	 At the moment my main concern is the proxy getting dos'd
[19:10:39] <musikanimal>	 I highly doubt it's a real attack, just bots gone wild
[19:10:45] <musikanimal>	 XTools uses the standard proxy
[19:11:43] <andrewbogott>	 ok. That makes me think the new proxy code is more vulnerable to this
[19:12:05] <andrewbogott>	 I guess when I say 'attack' I'm not assuming malice; the result is the same from malice or incompetence :)
[19:12:27] <musikanimal>	 yup, exactly
[19:12:46] <musikanimal>	 I think the Anubis upgrade was the main thing that did the trick for XTools
[19:13:56] <andrewbogott>	 Do you know who is still active in wikisource?
[19:15:27] <musikanimal>	 Samwilson is the one to talk to, he's surely asleep at the moment. He usually starts his day sometime after 00:00 UTC
[19:18:17] <andrewbogott>	 ok, ty
[19:18:57] <andrewbogott>	 Would it ruin the world if I just closed down that proxy for the moment?
[19:19:33] <musikanimal>	 no, it goes down a lot, lol
[19:20:31] <musikanimal>	 it is quite important to the Wikisource community so hopefully it gets resolved sooner than later, but folks are used to it being down from time to time
[19:25:36] <andrewbogott>	 !log wikisource stopping apache2 on wsexport-main01.wikisource.eqiad1.wikimedia.cloud
[19:25:38] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikisource/SAL
[19:31:17] <andrewbogott>	 musikanimal: can you spare a few more minutes to properly tag T437349 and also add links and/or responses about your mitigation thoughts?
[19:31:17] <stashbot>	 T437349: wsexport traffic issues - https://phabricator.wikimedia.org/T437349
[19:31:30] <musikanimal>	 sure thing!
[19:31:43] <andrewbogott>	 thank you!
[19:35:48] <Southparkfan>	 !log deployment-prep set profile::mediawiki::httpd::enable_forensic_log to false, to avoid disk exhaustion during high-traffic load
[19:35:51] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL
[19:36:27] <Southparkfan>	 !log deployment-prep truncate Apache2 forensic log on deployment-mediawiki14, disk almost full
[19:36:29] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL