[01:09:32] PROBLEM - MariaDB sustained replica lag on m1 on db2078 is CRITICAL: 15.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2078&var-port=13321 [01:11:40] RECOVERY - MariaDB sustained replica lag on m1 on db2078 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2078&var-port=13321 [07:33:57] Amir1: It seems that https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/758715/1/wmf-config/db-production.php hasn't disabled writes on es4 [07:34:02] I can still see them coming thru [07:34:44] you synced the wrong file [07:34:56] I did? [07:35:00] taavi: oh that was fast [07:35:02] thanks [07:35:09] ah yes! I am so used to the other one [07:35:10] marostegui: use deploy commands [07:35:35] taavi: Syncing now the good one, thanks for that catch [07:35:42] https://deploy-commands.toolforge.org/bacc/758715 [07:35:43] Amir1: What are deploy commands? We only use scap-file [07:36:03] Amir1: THat's what I used I simply did a crlt+r :) [07:36:11] And we normally only deploy the other file [07:36:29] this gives you exact commands to use for deployment (and formatted with proper ticket number, etc.) [07:36:59] marostegui: I know but this is to reduce the chance of mistakes happening :D [07:37:16] That tool is useful though! Bookmarked :) [07:37:25] Writes are stopped now :) [08:51:58] And my laptop died [08:52:26] It'll take a while to find a place to connect it to a charger [08:57:03] * Emperor is around, but expecting the es4 switchover to be routine? [08:59:37] Going to start it now, if you want to follow go to -operations [09:08:58] Amir1: i don't know of any api for dbctl. [09:10:45] I think there are ?spicerack? functions for pooling/depooling/querying pool status/ [09:12:33] Emperor: and that's dbctl, and not conftool? [09:13:54] * Emperor might be confused [09:14:30] the only way to interact with dbctl that i know of is to run the dbctl binary. it manipulates files on disk, and then data in etcd [09:14:46] it does not lend itself to being used as a library [09:17:19] Amir1: look at conftool.extensions.dbconfig in operations/software/conftool.git [09:17:40] there are zero matches for the string 'dbctl' in the spicerack+cookbooks repos [09:18:39] so, no, i'm pretty confident there's no automation around this. [09:20:56] jynus: ok to stop mysql on es1021? (old master) [09:21:24] I belive yes, but if you don't mind me taking a second to double check? [09:21:33] go for it! [09:22:28] marostegui: look ok, both es backups finished correctly yesterday, go ahead [09:22:36] thanks! [09:31:36] volans: I am running the provisioning script for es1021 to fix the PXE stuff :) [09:31:43] yay [09:31:48] hope it works :D [09:32:01] last one of the batch? [09:32:10] no, there's also es5 current master left [09:32:12] do you want me to check more extensively other hosts? [09:32:16] once we switch it over, we'll do that [09:32:19] k [09:33:02] volans: it wouldn't hurt of course, but that specific misconfig seems related to that last batch [09:33:17] I haven't found those issues before and I have reimaged quite a bunch of other batches host [09:33:19] hosts [09:33:27] ack, thanks [09:51:26] volans: looks like it worked [09:51:28] :) [09:51:37] glad it did :) [10:38:49] I have removed recentchanges* from eqiad s4, which are big groups, please let me know if you see something unexpected traffic-wise [14:34:54] godog: while poking around the swift fe setup (you have a CR about bringing new proxies online), I noticed that the codfw nodes are weight 10 for service=nginx whereas the eqiad nodes are weight 40. What's the rationale for that difference? And what is the distinction between the nginx and swift-fe services in etcd, please? (they appear in pybal as swift and swift-https) [14:36:21] [I wondered if it related to which cluster was rw, but the last switchover didn't adjust any weights AFACT] [15:16:18] hello! i'm need to ensure that a mariadb client package is present via puppet [15:16:24] which package should I use? [15:16:31] wmf-mariadb104-client, or just mariadb-client? [15:17:08] i think i see mariadb-client being used more often [15:17:29] I belive the second is the debian official one [15:17:40] right [15:18:05] so that one? okay! [15:18:30] for the client, there shouldn't be many changes [15:18:33] ya [15:20:30] ottomata: i'd use wmf-mariadb104-client [15:21:10] ottomata: or, more precisely, profile::mariadb::packages_client [15:46:36] kormat: can i just use wmf-mariadb104-client? i don't think i want those other packages [15:52:03] Emperor: re: weight difference I think different defaults at the time of setup (i.e. weights can be equalized), re: nginx vs swift-fe I think it is for histerical raisins and the history of pybal vs conftool [15:52:15] I'll take a look at the CR! [16:03:15] Thanks. (if we can simplify to just swift-fe rather than that and nginx, that'd be nice :) [16:23:45] yeah I think that should be possible [16:29:23] ottomata: .. what other packages? [16:38:43] kormat: [16:38:43] https://github.com/wikimedia/puppet/blob/production/modules/mariadb/manifests/packages_client.pp [16:39:33] ottomata: ahh. i should have looked there, sorrry :) [16:41:15] ottomata: so, sure, you can just install wmf-mariadb104-client if that's all you want [16:44:02] Pooled ms-fe2009, things look stable, so going to now also add 10-12 [16:51:03] godog: new frontends look in service and OK to me, no obvious 🔥... [16:51:04] ok great, thank you [16:52:36] Emperor: nicely done! can confirm lack of 🔥 [16:53:44] 😌