[07:16:05] <jynus>	 Could I get a green light on the plan @ https://gerrit.wikimedia.org/r/c/operations/puppet/+/1324577 after puppet compiler runs there?
[07:49:20] <marostegui>	 federico3: can you check the private data emails please?
[07:50:13] <federico3>	 ok
[08:18:04] <federico3>	 @marostegui it's a table from testwiki - I'm looking for related tasks but this is not a new wiki, should I run the sanitization anyways?
[08:19:38] <marostegui>	 what's the table name?
[08:35:32] <federico3>	 cx_corpora
[08:35:53] <marostegui>	 I know what that comes from: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1324309
[08:36:03] <marostegui>	 so yeah, please run the sanitization across all hosts that reported via email
[08:40:34] <jynus>	 I'm having issues with my pc
[08:41:14] <jynus>	 I asked for a work laptop but they told me to wait until after my vacations
[08:55:16] <jynus>	 db2250 will take a bit to catch up, as I have recovered an old backup
[08:56:30] <jynus>	 it will need schema changes applied to s5 from 2026-08-05 20:00:16 to 2026-08-12 08:56:12
[08:59:39] <marostegui>	 jynus: there wasn't any I think
[09:00:23] <jynus>	 still something I wanted to be checked to avoid issues
[09:00:30] <jynus>	 or at least wanted to notify
[09:00:45] <jynus>	 I am adding it to zarcillo now
[09:01:09] <jynus>	 it will be with large lag on orchestrator for a long time
[09:01:37] <jynus>	 I will finish the productization and pool it as backup source
[09:17:19] <jynus>	 I just used the downtime feature in orcherstator for the first time
[09:27:47] <jynus>	 it will take around 6h to catch up: https://grafana.wikimedia.org/goto/cfuy2pw6wui9se?orgId=default
[09:35:41] <cezmunsta>	 jynus: please can you comment on https://phabricator.wikimedia.org/T434635 once it has caught up so that I can go ahead and do the switchover
[09:36:35] <jynus>	 sure
[09:36:45] <cezmunsta>	 +1
[09:36:48] <jynus>	 it will be at the end of the day
[09:36:54] <jynus>	 today, based on estimations
[09:37:26] <jynus>	 or you can self serve based on the pasted graph in case I am not around 0:-), so you don't get blocked on me
[09:37:40] <jynus>	 but I will comment as soon as that happens
[09:38:34] <cezmunsta>	 Thanks
[09:58:44] <cezmunsta>	 Doing s1 codfw switchover
[10:47:08] <jynus>	 ok to go with https://gerrit.wikimedia.org/r/c/operations/puppet/+/1324672 ?
[10:53:46] <cezmunsta>	 Wouldn't it be better to make sure that it catches up first?
[11:10:50] <jynus>	 ideally yes, but I need to test it early or it will become "your problem", as I will be going on vacations on friday 0:-D
[11:11:23] <jynus>	 that's why I would prefered deployed and tested asap
[11:24:46] <jynus>	 the alternative is, ofc, keep backing up a potentially corrupt instance
[11:25:15] <jynus>	 backups are not like mediawiki, they don't care much about the real time of the db
[13:17:28] <jynus>	 db2250:s5 still has approximatelly 2 hours until catch up
[13:57:51] <federico3>	 @marostegui the sanitization script is failing to connect to clouddb1029.eqiad.wmnet - the host is running but it has s5 and s8 and the sanitization is for s3
[13:59:03] <federico3>	 have you seen this happening before?
[14:00:45] <cezmunsta>	 federico3: do you mean the redact script?
[14:02:52] <federico3>	 the sanitize-wiki cookbook
[14:03:08] <cezmunsta>	 OK, you said script 
[14:05:06] <cezmunsta>	 What was the full command that you used?
[14:13:27] <cezmunsta>	 federico3: isn't this the issue? self.clouddb_hosts = self.mysql.get_dbs("A:db-clouddb-sanitization")
[14:17:41] <cezmunsta>	 Compare to "A:db-section-s3 and P{clouddb*}" for example
[14:22:04] <federico3>	 I think we often used sanitize-wiki with --section s5 so it would not be an issue but we also used it for s3 and perhaps it should detect if the target section is present otherwise skip it
[14:23:20] <cezmunsta>	 The code looks like it attempts all clouddb hosts rather than only those for the santarium hosts, which is what I was pointing to with the cumin alias
[14:24:30] <cezmunsta>	 i.e. compared to "A:db-section-<section> and (P{clouddb*} or P{an-redactedd*})"
[15:05:06] <jynus>	 cezmunsta: do you plan to do the s5 codfw switchover today?
[15:06:13] <jynus>	 because if not, I would like to run some backup testing on db2250
[15:06:28] <jynus>	 and tomorrow you have it all for you
[15:06:57] <jynus>	 alternativelly, I can wait until tomorrow, whatever you prefer
[15:34:12] <jynus>	 I started a dump now, should finish in 1h or so
[15:47:13] <cezmunsta>	 jynus: I am OK to wait until done with the backups
[15:48:08] <jynus>	 it actually should finish faster, in like less than 20 minutes.
[15:49:05] <jynus>	 this way, the long term backups pick the good one rather than the bad one from tuesday
[16:09:21] <jynus>	 backup finished
[16:09:24] <jynus>	 all free
[16:50:01] <cezmunsta>	 Thanks :)
[16:53:22] <cezmunsta>	 Doing s5 codfw switchover