[05:57:49] <marostegui>	 Going to bring up db1115 to try to clean up tendril DB before the reimage
[05:58:39] <marostegui>	 I need to remember to start it with events disabled, otherwise it will be fun...
[07:28:38] <marostegui>	 https://phabricator.wikimedia.org/T297605#7699852
[07:29:22] <marostegui>	 I am going to spend time now trying to clean up tendril stuff from puppet
[09:01:13] <marostegui>	 jynus: Just checking, nothing on the backups side relays on db1115 having the tendril DB right? I don't think so but just double checking
[09:01:40] <jynus>	 I belive so, but let me think
[09:01:59] <jynus>	 well, backups of zarcillo happen there (on both eqiad and codfw)
[09:02:17] <marostegui>	 yeah that's not a problem. I mean that the tendril DB is gone
[09:02:58] <jynus>	 no, section checks are manually configured- as we didn't have yet a good source of truth
[09:03:12] <marostegui>	 cool
[09:03:35] <jynus>	 and same for backups themselves- the list of hosts are all manually setup
[09:04:22] <jynus>	 what's the status now to update zarcillo backups? Would a new host be setup for zarcillo, reuse the same?
[09:05:14] <marostegui>	 jynus: that hasn't changed
[09:05:19] <jynus>	 ok
[09:05:39] <marostegui>	 Zarcillo remains where it was - db1115 was reimaged keeping the data
[09:05:45] <jynus>	 cool
[09:05:48] <marostegui>	 and db2093 hasn't been touched
[09:05:52] <marostegui>	 (yet)
[09:06:25] <jynus>	 then that is all on my side, the only possibility is if there is some unknown dependency of wmfmariadbpy, which the backups depend on
[09:07:02] <jynus>	 but my guess would be that, other than the tendril update on switchover, there won't be much
[09:07:12] <marostegui>	 cool then!
[09:07:14] <marostegui>	 thanks for checking
[09:08:20] <jynus>	 I see, there is some doc reference to db1115, but that is not important
[09:09:04] <marostegui>	 yeah, I have been cleaning up db1115 and tendril references on puppet and other repos
[09:09:09] <marostegui>	 but there're quite a few still
[09:09:14] <jynus>	 help=('Section name of the backup. E.g.: "s3", "tendril"
[09:09:17] <marostegui>	 I would assume wikitech needs also some cleaning
[09:09:35] <jynus>	 but it is not too missleading to require immediate action
[09:09:59] <marostegui>	 yeah no issue from my side
[09:29:30] <marostegui>	 I have removed watchdog@10.% user from db2071 to see if something complains
[09:29:36] <marostegui>	 let me know if someone see something weird
[11:05:04] <Amir1>	 marostegui: you will like this https://phabricator.wikimedia.org/T301433#7700220
[11:05:11] <Amir1>	 https://grafana.wikimedia.org/d/000000273/mysql?viewPanel=2&orgId=1&refresh=1m&var-job=All&var-server=db1118&var-port=9104
[11:06:23] <Amir1>	 even traffic to master got smaller https://grafana.wikimedia.org/d/000000273/mysql?viewPanel=5&orgId=1&refresh=1m&var-job=All&var-server=db1118&var-port=9104
[11:07:55] <marostegui>	 oh nice!
[11:19:18] <marostegui>	 Amir1: you are not doing anything on s2, right?
[11:19:43] <Amir1>	 marostegui: I'm doing bullseye upgrade there https://phabricator.wikimedia.org/T300510
[11:19:49] <marostegui>	 ah ok
[11:19:50] <Amir1>	 but it can wait for a day or two
[11:19:54] <marostegui>	 Nah
[11:20:00] <marostegui>	 I will move to s8
[11:20:02] <marostegui>	 or s7
[11:20:03] <marostegui>	 no worries
[11:20:21] <Amir1>	 cool
[11:20:30] <Amir1>	 I'm doing a schema change on s1 as well
[11:20:44] <marostegui>	 Sure, no problem
[11:20:46] <marostegui>	 I am taking s7
[13:02:13] <Amir1>	 marostegui: is it fine if I "decom" the tendril VM?
[14:07:11] <marostegui>	 Amir1: sure, if it's been stopped for a few days, and nothing broke, yeah
[14:07:19] <marostegui>	 that means tendril.wikimedia.org will be gone, right?
[14:07:30] <marostegui>	 ah no, that's being served from the static stuff
[14:07:31] <marostegui>	 right?
[14:08:31] <Amir1>	 yup
[14:08:35] <Amir1>	 that's miscweb now
[14:08:53] <Amir1>	 tendril-legacy.wikimedia.org would be gone but we already removed its DNS entry anyway
[14:09:07] <marostegui>	 yep
[14:09:27] <Amir1>	 I'm going to do the first bullseye upgrade on a multiinstance host
[14:09:30] <Amir1>	 fingers crossed
[14:37:05] <marostegui>	 nothing broke on db2071 after removing watchdog grant, I might remove it from a section later
[17:38:10] <marostegui>	 https://jira.mariadb.org/browse/MDEV-27789
[17:38:46] <marostegui>	 ^ that's why they are not releasing 10.4.23 anymore and moving directly to 10.4.24
[18:13:02] <volans>	 ouch