[01:10:05] PROBLEM - MariaDB sustained replica lag on m1 on db2160 is CRITICAL: 9.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321 [01:11:55] RECOVERY - MariaDB sustained replica lag on m1 on db2160 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321 [11:30:45] Hey peeps, I'd like a firm answer to https://phabricator.wikimedia.org/T329533 since it is currently blocking my datacenter switch tests, can I get a DBA stamp on the task please? [11:33:45] I was having an unhappy thought about swift in the small hours (as one does). Mediawiki attempts to write each new object to both ms swift clusters. It doesn't always succeed, and thus we have swiftrepl (and, lately, rclone). Swiftrepl assumes that the primary DC is correct (if X is in primary and not secondary, copy it from primary to secondary; if X is in secondary and not primary, delete it from secondary); is that a good assumpt [11:33:45] What does MW say to the user if one of its two write attempts fails? Does it matter which fails? [11:34:29] I have no idea :/ [11:36:59] * Emperor wonders if Amir1 knows [11:37:22] (but I have other stuff I'm meant to be unhappy about today) [11:37:29] good morning [11:39:30] responded there [11:39:38] fingers crossed I don't make a mess :D [11:46:03] Amir1: Thanks <3 [15:05:13] Emperor: about to merge, https://gerrit.wikimedia.org/r/c/operations/puppet/+/773298 , ok? [15:05:55] 👀 [15:06:06] let me do a puppet compiler run just in case [15:06:37] jynus: yeah, I'm content with that [15:08:50] looks good: https://puppet-compiler.wmflabs.org/output/773298/39678/ms-fe1009.eqiad.wmnet/index.html [15:12:24] Emperor: I saw what looks like some leftovers of a password migration on labs/private FYI [15:13:20] how do you mean? [15:13:48] swift::params::account_temp_url_keys [15:14:12] (maybe they are there for a reason, but they are not on production) [15:14:41] on labs/private: hieradata/eqiad/swift/params.yaml [15:14:42] git blame suggests they date from 2019 [15:14:53] oh, so quite old [15:15:06] then no issue, leftover to delete at some point [15:15:12] *leftovers [15:15:26] that's my feeling, but Chesterton's Fence applies, so I'm probably not going to touch them any time soon :) [15:15:37] we have many of those indeed don't worry [15:15:53] I thought they were from a recent refactoring you may know [15:16:06] not guilty ;-) [15:16:12] (at least this time!) [15:21:07] I will now modify the client code to use those same credentials [15:37:01] I get a 401 Unauthorized when running commands from ms-fe1009 with the new credentials [15:39:04] do we need to do a rolling restart? (I didn't see any reload on puppet run) [15:49:24] we do [16:00:36] jynus: yes, rolling-restart is required when changing any swift credential [16:00:57] I gotcha, it requires rolling and that is why it is not done by puppet [16:01:08] those basic things I am finding out now :-P