|
2024-08-07 08:44:12
|
<dcaro>
|
!status some instability due to storage rebalancing
|
|
2024-08-07 08:57:04
|
<dcaro>
|
!log tools rebooting a first batch of k8s nfs workers that have nfs stuck processes
|
|
2024-08-07 13:15:11
|
<dhinus>
|
!log tools.stashbot ./bin/stashbot.sh restart # a user reported in IRC it was not working
|
|
2024-08-07 13:15:12
|
<stashbot>
|
dhinus: Failed to log message to wiki. Somebody should check the error logs.
|
|
2024-08-07 13:17:58
|
<dhinus>
|
hmm stashbot is failing to write to wiki apparently, I can see a few errors in the logs
|
|
2024-08-07 13:18:35
|
<dhinus>
|
bd808: can you please have a look when you are online?
|
|
2024-08-07 13:21:02
|
<dcaro>
|
!log tools testing stashbot
|
|
2024-08-07 13:21:03
|
<stashbot>
|
dcaro: Failed to log message to wiki. Somebody should check the error logs.
|
|
2024-08-07 13:30:31
|
<wm-bb>
|
<lucaswerkmeister> very detailed error messages :/
|
|
2024-08-07 13:30:53
|
<wm-bb>
|
<lucaswerkmeister> “NoWriteApi”, “Error writnig to wiki”. not seeing anything useful-looking in stashbot.log so far
|
|
2024-08-07 13:32:46
|
<wm-bb>
|
<lucaswerkmeister> general editing seems to work on wikitech and I’m not seeing anything in the abuse log either
|
|
2024-08-07 13:32:52
|
<dhinus>
|
yep that same error is repeated every time someone tries to log something
|
|
2024-08-07 13:33:37
|
<dhinus>
|
maybe an authentication problem?
|
|
2024-08-07 13:36:50
|
<wm-bb>
|
<lucaswerkmeister> ohhh nooooo
|
|
2024-08-07 13:36:52
|
<wm-bb>
|
<lucaswerkmeister> I think I know what it is
|
|
2024-08-07 13:37:24
|
<wm-bb>
|
<lucaswerkmeister> https://en.wikipedia.org/w/api.php?action=query&meta=siteinfo has `"writeapi": ""`
|
|
2024-08-07 13:37:26
|
<wm-bb>
|
<lucaswerkmeister> https://wikitech.wikimedia.org/w/api.php?action=query&meta=siteinfo does not
|
|
2024-08-07 13:37:31
|
<wm-bb>
|
<lucaswerkmeister> because the right was removed from core
|
|
2024-08-07 13:37:45
|
<wm-bb>
|
<lucaswerkmeister> but mwclient expects this https://github.com/mwclient/mwclient/blob/8c4b3838f953fc54d819dc74c9a3d1e587b76de4/mwclient/client.py#L214-L220
|
|
2024-08-07 13:37:50
|
<wm-bb>
|
<lucaswerkmeister> so now it thinks this is a read-only site :(
|
|
2024-08-07 13:38:07
|
<wm-bb>
|
<lucaswerkmeister> I think that’s worth an UBN and train rollback tbh, SAL is pretty important
|
|
2024-08-07 13:38:12
|
<wm-bb>
|
<lucaswerkmeister> (I assume this also affects the prod SAL)
|
|
2024-08-07 13:38:24
|
<wm-bb>
|
<lucaswerkmeister> /me makes task
|
|
2024-08-07 13:42:19
|
<wm-bb>
|
<bd808> Thanks for looking @lucaswerkmeister and yes it would effect prod sal equally. And probably every other tool I have that writes to a wiki
|
|
2024-08-07 13:42:27
|
<dcaro>
|
thanks lucaswerkmeister
|
|
2024-08-07 13:43:33
|
<wm-bb>
|
<lucaswerkmeister> made T371977
|
|
2024-08-07 13:43:42
|
<wm-bb>
|
<lucaswerkmeister> (https://phabricator.wikimedia.org/T371977)
|
|
2024-08-07 13:47:33
|
<andrewbogott>
|
lucaswerkmeister, should we address that by changing mwclient instead? Or in addition?
|
|
2024-08-07 13:52:38
|
<wm-bb>
|
<lucaswerkmeister> cross-posted at https://github.com/mwclient/mwclient/issues/344 as well
|
|
2024-08-07 13:53:02
|
<wm-bb>
|
<lucaswerkmeister> @andrewbogott: even if mwclient is very fast to update (I don’t know who maintains it), all affected users will still have to update their environments to get the new version
|
|
2024-08-07 13:53:12
|
<wm-bb>
|
<lucaswerkmeister> I think this needs a fix in production first
|
|
2024-08-07 13:53:12
|
<andrewbogott>
|
yeah
|
|
2024-08-07 13:53:19
|
<andrewbogott>
|
agreed, I think we need to do both
|
|
2024-08-07 13:57:49
|
<andrewbogott>
|
lucaswerkmeister: naive patch (naive because I've never touched that code before) https://github.com/mwclient/mwclient/pull/345
|
|
2024-08-07 13:57:57
|
<andrewbogott>
|
and also 100% untested :)
|
|
2024-08-07 14:15:30
|
<wm-bb>
|
<lucaswerkmeister> (this is the first time I’ve seen mwclient btw, I usually use mwapi. so don’t count on me to merge anything there ^^)
|
|
2024-08-07 15:41:30
|
<wm-bot>
|
!log bd808@tools-bastion-12 tools.stashbot Installed a hacked up version of mwclient from git+https://github.com/bd808/mwclient@2e9cd61d90738fbf9ee64ca2c1766a9095c24699 to work around T371977
|
|
2024-08-07 15:41:33
|
<stashbot>
|
Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL
|
|
2024-08-07 17:28:42
|
<Joutbis>
|
Hello. I have a bot running on toolforge, and it has started raising NoWriteApi. Any idea on what may have happened?
|
|
2024-08-07 17:29:48
|
<wm-bb>
|
<lucaswerkmeister> that'll be the same error as T371977 then
|
|
2024-08-07 17:31:41
|
<Joutbis>
|
Yes, looks like it. I'll have to subscribe to that
|
|
2024-08-07 17:33:27
|
<bd808>
|
Joutbis: the rollback of the MediaWiki change that caused that bug is in progress right now. Hopefully we can keep it from getting broken again for a bit to give clients a chance to stop checking that right.
|
|
2024-08-07 17:59:46
|
<Joutbis>
|
OK, thanks a lot
|
|
2024-08-07 20:26:13
|
<mutante>
|
andrewbogott: any known maintenance happening in cloud by any chance?
|
|
2024-08-07 20:30:41
|
<andrewbogott>
|
mutante: what are you seeing?
|
|
2024-08-07 20:31:08
|
<andrewbogott>
|
We have a bad switch so are migrating lots of things way from it. In theory that's not disruptive
|
|
2024-08-07 20:31:33
|
<mutante>
|
andrewbogott: suddenly spike in traffic but only on management switches
|