[05:56:42] <marostegui>	 Thanks for the heads up Jdlrobson. From what I can see the script also uses wait for replication, so that's good. If 800 rows will be the max you'll be inserting/updating, that should be no problem. 
[06:57:52] <marostegui>	 Amir1:  /srv/mediawiki-staging/multiversion/MWScript.php extensions/FlaggedRevs/maintenance/pruneRevData.ph will refresh the config itself?
[06:58:07] <marostegui>	 Not sure if that's your script that's why I ask
[06:58:28] <marostegui>	 If it won't and it is hard to stop, no problem, I can run something there manually
[07:02:24] <marostegui>	 ok so it timeouted so I will run the change there manually, it is just one host ,so not a big deal
[07:25:18] <Amir1>	 marostegui: it doesn't 😔
[07:25:31] <marostegui>	 np!
[07:26:03] <Amir1>	 It should finish soon hopefully 
[08:07:33] <Amir1>	 marostegui: I have this one for you too to review https://gerrit.wikimedia.org/r/c/operations/puppet/+/770890 :P
[08:08:08] <marostegui>	 ok
[09:22:50] <Amir1>	 marostegui: Error connecting to db1099:3318 as user wikiadmin: :real_connect(): (HY000/2002): Connection refused 
[09:22:53] <Amir1>	 Is it normal?
[09:23:26] <marostegui>	 yes, it is depooled
[09:23:34] <marostegui>	 but I guess some script isn't reloading?
[09:23:41] <Amir1>	 some stuff are connecting to it :D
[09:23:51] <Amir1>	 I don't know what but it's triggering errors
[09:23:57] <marostegui>	 but if it is wikiadmin it must be a script
[09:24:00] <marostegui>	 https://phabricator.wikimedia.org/P22670
[09:24:14] <Amir1>	 it's the actor migration
[09:24:22] <marostegui>	 it is up now, but I am going to be testing it so it will be down in a sec again :)
[09:24:33] <Amir1>	 it should reload the config in ten minutes
[09:24:41] <marostegui>	 I am with https://phabricator.wikimedia.org/T303498
[09:24:50] <Amir1>	 noted
[10:20:34] <marostegui>	 Going to alter templatelinks on s2 master, it is going to take a few hours
[10:44:39] <marostegui>	 I am going to merge https://gerrit.wikimedia.org/r/c/operations/software/+/771334/ if no objections
[10:44:58] <marostegui>	 We are not going to do any switchovers before the 28th, so we can generate the news one with that removed
[12:38:22] <Emperor>	 https://grafana.wikimedia.org/d/OPgmB1Eiz/swift?orgId=1&var-DC=codfw&var-prometheus=codfw%20prometheus%2Fops&from=1647412677905&to=1647434217905&viewPanel=8 is not as helpful a Y-axis as it is possible to conceive of...
[14:25:48] <Emperor>	 going to depool ms-fe100[5-8] now, since ms-fe10[09-12] have been in service since Monday with no drama
[14:32:08] <marostegui>	 Altering commonswiki.templatelinks is going to be fun.....almost 300GB
[15:04:59] <marostegui>	 downtime_hours = 72
[15:05:02] <marostegui>	 this seems about right for that
[15:39:03] <marostegui>	 Amir1: For https://phabricator.wikimedia.org/T297189 we'll also need to clean up views references, I am going to clean up s5, so it is not broken now, and later I will create a task for that for the data team
[15:39:35] <Amir1>	 yeah awesome
[15:40:09] <Amir1>	 marostegui: keep in mind that once we start removing the non-normalized data from commonswiki.templatelinks, it'll get much smaller 
[15:41:14] <marostegui>	 Fixed s5, actually, I won't create the task and I will do it myself as soon as we progress on the ticket, as it will require coordination between you running the clean up and me doing all the changes
[15:41:46] <marostegui>	 it will be easier if I simply do it
[16:41:34] <Emperor>	 eqiad swift service now just running on the new proxies
[16:43:40] <jynus>	 higher os version?
[16:44:03] <Emperor>	 🙊
[16:44:08] <jynus>	 ha ha
[16:44:13] <Emperor>	 just newer hardware
[16:45:07] <jynus>	 I know you keep saying that Debian Hamm is very stable but...
[16:46:35] <Emperor>	 I spent too much time at my last place doing upgrades of iRODS servers from ancient ancient Ubuntus (or a few old RHEL systems). I was obviously very bad in a past life...
[16:47:13] <Emperor>	 seriously, though, I suspect much of my Q4 is going to be ms bullseye upgrades. I think go.dog has done some of the thanos swift nodes without overmuch pain now
[16:49:13] <jynus>	 https://i.imgflip.com/68w2w5.jpg
[16:51:23] <RhinosF1>	 Emperor: Ms just alerted if you're not expecting it
[16:51:34] <RhinosF1>	 Not ms, Swift
[16:53:17] <Emperor>	 RhinosF1: if you mean the Number of mw swift objects in codfw greater than eqiad alert, yes, it's a consequence of moving the stats_reporter_host and should self-resolve in 10m or so
[16:53:34] <RhinosF1>	 Emperor: ah
[16:53:39] <RhinosF1>	 Yes that
[16:54:27] <Emperor>	 Yeah, I was much more alarmed when that happened while I was moving the codfw stats reporter host the other week; this time its expected :)
[16:54:44] <RhinosF1>	 :)
[20:55:06] <cjming>	 hi all - ok if we run a maintenance script in a few minutes?
[20:55:34] <Jdlrobson>	 we'll be updating around 35 rows.
[20:56:37] <RhinosF1>	 Jdlrobson: in total or per wiki?
[20:56:48] <Jdlrobson>	 in total
[20:57:02] <Jdlrobson>	 hewiki and frwiki
[20:57:24] <Jdlrobson>	 dry run tells us 24 rows will be updated on Hebrew and 11 on French
[20:57:33] <RhinosF1>	 I can't imagine anyone would have even noticed tbh
[21:17:14] <cjming>	 !log end running skin update preference maintenance script
[21:17:14] <stashbot>	 cjming: Not expecting to hear !log here