[05:22:04] <marostegui>	 I am going to try again to clean gtid_domain_id on db_inventory
[06:11:51] <marostegui>	 looks like I got it right this time
[06:14:07] <marostegui>	 I am going to go for m5
[07:03:04] <marostegui>	 I broke it, then fixed it again, and then broke it again 
[07:03:07] <marostegui>	 This is so painful
[07:03:25] <marostegui>	 I am going to need to reclone codfw entirely
[07:11:15] <jynus>	 you should coordinate with Amir if ever tried in mediawiki, I think the chronology protector does some assumptions re:gtid
[07:11:32] <marostegui>	 yeah, there's lots of work there
[07:11:43] <marostegui>	 I really want to disable the pt-heartbeat on the secondary dc
[07:11:47] <marostegui>	 I need to talk to amir about it
[07:23:25] <marostegui>	 ok all fixed and codfw recloned
[07:45:52] <marostegui>	 I am rebooting dbproxy2004 (m5 codfw proxy) which is in a very weird state after all these issues
[07:45:56] <marostegui>	 and not detecting any backedn
[07:48:37] <marostegui>	 Ah, I know what it is
[07:50:04] <marostegui>	 fixed
[13:11:46] <marostegui>	 it only took me 4 hours to understand what went wrong XD
[13:13:29] <Emperor>	 fun times :)
[14:25:42] <urandom>	 Emperor: if swift returns a 400, does it log the underlying reason somewhere (and if so, where)?
[14:25:50] <urandom>	 Emperor: for example: https://logstash.wikimedia.org/app/discover#/doc/0fade920-6712-11eb-8327-370b46f9e7a5/ecs-k8s-1-1.11.0-6-2023.21?id=7LGvTIgBs53OSt3dsZkq
[15:01:19] <Emperor>	 urandom: swift's logging isn't very helpful. You might find something by grepping for the relevant timestamp in the frontends' proxy-access.log and server.log
[15:01:43] <Emperor>	 urandom: that's presumably thanos, so at least it's not so many frontends to go grepping in
[15:02:03] <urandom>	 it doesn't log remotely?
[15:02:25] <Emperor>	 I don't think we ship swift logs anywhere, no
[15:05:28] <Emperor>	 [if you would like more help/advice/heckling do shout]
[15:20:31] <gmodena>	 urandom re: T330693. I wonder if the issue is that the new bucket name contains '_'. 
[15:20:32] <stashbot>	 T330693: Storage request: swift s3 bucket for mediawiki-page-content-change-enrichment checkpointing - https://phabricator.wikimedia.org/T330693
[15:20:55] <gmodena>	 might be an invalid character for the s3 protocol https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html
[15:21:12] <gmodena>	 investigating 
[15:21:15] <gmodena>	 ^ ottomata 
[15:25:51] <gmodena>	 urandom ottomata replacing `_` with `-` did the trick, the checkpoints are stored as expected
[15:26:34] <urandom>	 gmodena: ha, I was just coming here to suggest the same
[15:28:20] <urandom>	 it works for swift (I guess that's how you create the container)
[15:28:27] <urandom>	 s/create/created/
[15:30:18] <gmodena>	 urandom indeed. I created the bucket with `swift post`
[15:30:36] <gmodena>	 urandom thanks for looking into this, and apologies for the noise.
[15:30:46] <gmodena>	 s/bucket/container
[15:30:54] <urandom>	 gmodena: no worries
[15:33:49] <Emperor>	 in s3 it's a bucket ;-)
[15:53:42] <urandom>	 right, it was a container that he created, one that did not work as a bucket :)
[16:43:17] <ottomata>	 i guess a bucket is a container but a container is not necessarily a bucket :p