[08:44:12] <dcaro>	 !status some instability due to storage rebalancing
[08:57:04] <dcaro>	 !log tools rebooting a first batch of k8s nfs workers that have nfs stuck processes
[13:15:11] <dhinus>	 !log tools.stashbot ./bin/stashbot.sh restart # a user reported in IRC it was not working
[13:15:12] <stashbot>	 dhinus: Failed to log message to wiki. Somebody should check the error logs.
[13:17:58] <dhinus>	 hmm stashbot is failing to write to wiki apparently, I can see a few errors in the logs
[13:18:35] <dhinus>	 bd808: can you please have a look when you are online?
[13:21:02] <dcaro>	 !log tools testing stashbot
[13:21:03] <stashbot>	 dcaro: Failed to log message to wiki. Somebody should check the error logs.
[13:30:31] <wm-bb>	 <lucaswerkmeister> very detailed error messages :/
[13:30:53] <wm-bb>	 <lucaswerkmeister> “NoWriteApi”, “Error writnig to wiki”. not seeing anything useful-looking in stashbot.log so far
[13:32:46] <wm-bb>	 <lucaswerkmeister> general editing seems to work on wikitech and I’m not seeing anything in the abuse log either
[13:32:52] <dhinus>	 yep that same error is repeated every time someone tries to log something
[13:33:37] <dhinus>	 maybe an authentication problem?
[13:36:50] <wm-bb>	 <lucaswerkmeister> ohhh nooooo
[13:36:52] <wm-bb>	 <lucaswerkmeister> I think I know what it is
[13:37:24] <wm-bb>	 <lucaswerkmeister> https://en.wikipedia.org/w/api.php?action=query&meta=siteinfo has `"writeapi": ""`
[13:37:26] <wm-bb>	 <lucaswerkmeister> https://wikitech.wikimedia.org/w/api.php?action=query&meta=siteinfo does not
[13:37:31] <wm-bb>	 <lucaswerkmeister> because the right was removed from core
[13:37:45] <wm-bb>	 <lucaswerkmeister> but mwclient expects this https://github.com/mwclient/mwclient/blob/8c4b3838f953fc54d819dc74c9a3d1e587b76de4/mwclient/client.py#L214-L220
[13:37:50] <wm-bb>	 <lucaswerkmeister> so now it thinks this is a read-only site :(
[13:38:07] <wm-bb>	 <lucaswerkmeister> I think that’s worth an UBN and train rollback tbh, SAL is pretty important
[13:38:12] <wm-bb>	 <lucaswerkmeister> (I assume this also affects the prod SAL)
[13:38:24] <wm-bb>	 <lucaswerkmeister> /me makes task
[13:42:19] <wm-bb>	 <bd808> Thanks for looking @lucaswerkmeister and yes it would effect prod sal equally. And probably every other tool I have that writes to a wiki
[13:42:27] <dcaro>	 thanks lucaswerkmeister
[13:43:33] <wm-bb>	 <lucaswerkmeister> made T371977
[13:43:42] <wm-bb>	 <lucaswerkmeister> (https://phabricator.wikimedia.org/T371977)
[13:47:33] <andrewbogott>	 lucaswerkmeister, should we address that by changing mwclient instead?  Or in addition?
[13:52:38] <wm-bb>	 <lucaswerkmeister> cross-posted at https://github.com/mwclient/mwclient/issues/344 as well
[13:53:02] <wm-bb>	 <lucaswerkmeister> @andrewbogott: even if mwclient is very fast to update (I don’t know who maintains it), all affected users will still have to update their environments to get the new version
[13:53:12] <wm-bb>	 <lucaswerkmeister> I think this needs a fix in production first
[13:53:12] <andrewbogott>	 yeah
[13:53:19] <andrewbogott>	 agreed, I think we need to do both
[13:57:49] <andrewbogott>	 lucaswerkmeister: naive patch (naive because I've never touched that code before) https://github.com/mwclient/mwclient/pull/345
[13:57:57] <andrewbogott>	 and also 100% untested :)
[14:15:30] <wm-bb>	 <lucaswerkmeister> (this is the first time I’ve seen mwclient btw, I usually use mwapi. so don’t count on me to merge anything there ^^)
[15:41:30] <wm-bot>	 !log bd808@tools-bastion-12 tools.stashbot Installed a hacked up version of mwclient from git+https://github.com/bd808/mwclient@2e9cd61d90738fbf9ee64ca2c1766a9095c24699 to work around T371977
[15:41:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL
[17:28:42] <Joutbis>	 Hello. I have a bot running on toolforge, and it has started raising NoWriteApi. Any idea on what may have happened?
[17:29:48] <wm-bb>	 <lucaswerkmeister> that'll be the same error as T371977 then
[17:31:41] <Joutbis>	 Yes, looks like it. I'll have to subscribe to that
[17:33:27] <bd808>	 Joutbis: the rollback of the MediaWiki change that caused that bug is in progress right now. Hopefully we can keep it from getting broken again for a bit to give clients a chance to stop checking that right.
[17:59:46] <Joutbis>	 OK, thanks a lot
[20:26:13] <mutante>	 andrewbogott: any known maintenance happening in cloud by any chance?
[20:30:41] <andrewbogott>	 mutante: what are you seeing?
[20:31:08] <andrewbogott>	 We have a bad switch so are migrating lots of things way from it. In theory that's not disruptive
[20:31:33] <mutante>	 andrewbogott: suddenly spike in traffic but only on management switches