[08:06:21] <marostegui>	 jynus: are you using db1133 for anything? (mariadb::shard: 'test-s1') I want to take it and move it for test-s4 to test orchestrator recoveries (https://phabricator.wikimedia.org/T322993)
[08:06:32] <marostegui>	 moritzm: I am finished with cumin1001 btw
[08:10:29] <moritzm>	 excellent, then we should be all set for the reboot later
[08:21:23] <jynus>	 marostegui: not at the moment
[08:21:38] <marostegui>	 jynus: cool, it shouldn't take long (last famous words)
[10:06:59] <jynus>	 moritzm: let me know when reboot is done as I want to upgrade a package there
[10:07:54] <moritzm>	 you can go ahead, the cookbook is still active, but that's only because it's waiting for the keyholder to be fully armed again (needs netops to do it for the homer passphrase)
[10:08:10] <jynus>	 thanks, I will just do an apt install
[10:08:11] <moritzm>	 but Cumin etc. can be used already
[10:10:24] <jynus>	 there is a long list of upgradable packages there- I will only upgrade mine
[11:22:59] <marostegui>	 https://phabricator.wikimedia.org/P40749 this makes me cry
[11:51:14] <jynus>	 "just use GTID"- they said
[11:54:42] <_joe_>	 marostegui: LOL wth is that
[11:55:05] <marostegui>	 _joe_: your worst nightmare 
[11:55:34] <_joe_>	 your worst nightmare, I'd say :P
[12:02:51] <jynus>	 _joe_: I'd call it the definition of technical debt, except it is not create by but, but by someone else and cannot be removed 😭
[12:03:02] <jynus>	 *created by you
[12:03:40] <_joe_>	 I mean, I get it is complicated if you're doing multi-source replication
[12:03:52] <_joe_>	 which is in itself evil
[12:04:03] <_joe_>	 but otherwise, "how hard can it be"
[12:55:12] <marostegui>	 At least FLUSH BINARY LOGS DELETE_DOMAIN_ID (which was implemented after we filled the bug years ago) seems to be working nicely
[12:55:31] <jynus>	 oh, that's great!
[12:55:40] <marostegui>	 Yeah, so far it seems to be working
[12:55:45] <marostegui>	 But it is scary to run it
[12:56:19] <jynus>	 please document it on wikitech somewhere, even if it with a big //TODO / to be tested
[12:56:31] <jynus>	 it would impact my work on automation of recoveries too
[12:56:38] <marostegui>	 yeah
[12:56:39] <Emperor>	 Template: WCPGW?
[12:56:52] <marostegui>	 I am still testing it on the test cluster to recover things with GTID via orchestraror
[12:57:01] <marostegui>	 Emperor: no idea what that means XD
[12:57:15] <jynus>	 "what could possible go wrong"
[12:57:19] <Emperor>	 What Could Possibly Go Wrong
[12:57:19] <marostegui>	 ah
[12:57:20] <jynus>	 *bly
[12:57:21] <marostegui>	 XD
[12:57:41] <jynus>	 the fact that it exists is already a great step
[12:57:42] <marostegui>	 Then yes, not running that in production any time soon XD
[12:58:03] <marostegui>	 jynus: yeah, it was the workaround they implemented after the whole mess with gtid+multisource we found years ago
[12:58:52] <jynus>	 s/mess/whole fundamental design model and architecture problems/ buy yes
[12:58:57] <jynus>	 *but
[12:58:59] <marostegui>	 XD
[12:59:20] <marostegui>	 https://jira.mariadb.org/browse/MDEV-12012 wow, it was 2017 
[12:59:30] <marostegui>	 I didn't remember it was that long ago
[12:59:38] <jynus>	 what is your plan with orchestrator, btw (just curious) - gtid, binlog, pseudo-gtids?
[12:59:48] <marostegui>	 I would like to stick to gtid if possible
[12:59:53] <marostegui>	 If not, binlog
[13:00:35] <jynus>	 whatever works, I would say at this point :-D
[13:00:39] <marostegui>	 haha yeah
[13:01:57] <jynus>	 did you see our convo on -sre about hashing?
[13:02:21] <jynus>	 I think that shouldn't affect dbas much, that I can think of
[13:02:41] <jynus>	 but for us backups and media storage could have a lot of impact on performance
[13:02:51] <jynus>	 (in a good way)
[13:46:24] <marostegui>	 heh, using FLUSH BINARY LOGS DELETE_DOMAIN_ID is the way to go if we want to let orchestrator recover things via GTID from what I am testing now
[13:46:28] <marostegui>	 I send a summary later on the task
[13:46:40] <marostegui>	 but looks promising......we only have to go and clean it up _everywhere_ XD
[13:47:13] <marostegui>	 But otherwise orchestrator will pick the first domain_id from https://phabricator.wikimedia.org/P40749 which might or might not be the current master
[14:21:08] <jynus>	 :-(
[15:02:59] <Amir1>	 marostegui: db1132 in s1 has both /srv/sqldata and /srv/sqldata.s1 (while it's not multiinstance), that's confusing my poor restart script. I guess the .s1 one is from a transfer/clone. shall I delete it?
[15:03:54] <marostegui>	 might have been a left over yes
[15:04:01] <marostegui>	 can be deleted yes
[15:04:09] <Amir1>	 awesome
[15:04:29] <Amir1>	 I depool before deletion, just in case
[15:05:20] <Amir1>	 I hope I can get to automating provisioning and healing next Q
[15:08:01] <Amir1>	 the directory is last written in March, looks good for removal
[15:10:36] <Amir1>	 https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=db1132&var-datasource=thanos&var-cluster=wmcs&from=1669215583633&to=1669216222447&viewPanel=28 xD