[06:33:50] Just finished with all the parsercache failovers and reboots [09:04:53] heads up for dbstore1003 MariaDB disk space, btullis [09:06:06] the other ongoing alert relevant for this channel is a disk failure on ms-be2042 [09:06:39] I guess we need to do some optimizations on dbstore1003 [09:07:03] I haven't checked, could also be a staging db or something else [09:07:11] nah that one only has s1, s5 and s7 [09:07:32] Also I am wondering when dbstore needs to be replaced [09:07:36] let me see [09:07:57] Excellent, next fiscal! [09:13:33] I wonder if we should have daemons called wmf_auto_restart_wmf_auto_restart_prometheus-mysqld-exporter.service [09:13:46] given how much they fail [09:25:20] jynus: I think that's one of the ones due to be decommissioned shortly [09:25:34] /dev/sde1 3.7T -18M 3.7T 0% /srv/swift-storage/sde1 [09:25:34] [09:25:41] ^-- not sure I _quite_ believe that [09:27:24] it is ok, in that case just ack the alert or downtime it so it doesn't show up on the dashboard [09:44:02] s1 in dbstore1003 is way too large for the usual s1 dbs. I think some tables there need optimization [09:44:34] marostegui: I push 0.10 of mariadbpy, what is the next step then? https://debmonitor.wikimedia.org/packages/python3-wmfmariadbpy [09:44:47] go to cumin and do sudo apt-get update? [09:45:09] (sorry this is my first time packaging and releasing to the fleet) [09:46:08] Amir1: https://debmonitor.wikimedia.org/packages/python3-wmfmariadbpy [09:46:34] I literally linked to it :D [09:46:51] ah, sorry, I just read the last lines [09:51:02] Amir1: https://wikitech.wikimedia.org/wiki/Software_deployment [09:52:48] that's nice. Just double checking with Manuel since I'm sure we don't do with mariadb itself (and gradually roll it out) [10:25:45] for the backup hosts, I would just leave one host in the new version, in case you think it could lead to problems, and ugprade the following day [11:03:16] yeah, Amir1 just go to cumin2002 maybe and test there [11:03:22] We should test all the scripts just in case [11:03:33] Amir1: same with wmdb [11:03:46] wmfdb i meant [11:08:42] awesome [11:10:19] marostegui: I'm sorry I'm very pingy today, I'm about to start rebooting es1-3 (RO ones), beside backups that I'm taking into account, I should avoid doing "stop slave"/"start slave" there but otherwise, everything is the same? [11:19:39] Amir1: I would pick a different day, the switch operation happens in less than two hours [11:19:53] ah yeah [11:19:57] okay, tomorrow [11:20:16] so yeah, nothing else than that, but you need to switch masters, otherwise they won't let you depool the master [11:20:40] yeah, masters is going to be fun [14:05:57] Are cross-database queries also forbidden in prod? e.g. from enwiki -> USE metawiki; ? [14:07:04] you can't do cross-database joins. but mediawiki code can connect to different databases than the current wiki, for example for global preferences, userrights-interwiki or centralauth [14:07:53] I'm trying to figure out why a maintenance script is failing with Error 1049 from Wikimedia\Rdbms\DatabaseMysqlBase::doSelectDomain, Unknown database 'metawiki' USE `metawiki` db1184 [14:08:29] I know there are some restrictions in the replicas but I was not sure about prod [14:10:57] hmm: PHP Deprecated: Deprecated passing invalid cross-wiki user. Expected: the local wiki, Actual: 'enwiki'. [Called from MediaWiki\User\ActorStore::findActorId]PHP Deprecated: Deprecated passing invalid cross-wiki user. Expected: the local wiki, Actual: 'enwiki'. [Called from MediaWiki\User\ActorStore::findActorId] [14:12:09] so maybe they are not setting the $db correctly [14:12:44] you cannot do cross db queries mostly because they are on different physical machines [14:13:28] you have to ask for different connections [14:13:46] (and won't be able to join them) [15:43:28] codfw has drained the old hosts nicely; alas eqiad not so much, because of T336778 [15:43:29] T336778: Urgent: disk failed in ms-be1063 - https://phabricator.wikimedia.org/T336778