[05:24:36] <marostegui>	 federico3: We received again the private data alerts :(
[05:24:49] <marostegui>	 The script didn't finish?
[05:28:40] <marostegui>	 federico3: When you pinged me yesterday I was afk - the problem was that that table was in testwiki which is in s3. This is not a super common thing to happen as 95% of the cases where we need to redact something (a new wiki) it will live on the default section (s5), but for this issue, the table was present in test wiki, so we do need to redact that one.
[05:29:15] <marostegui>	 federico3: so you'll have to redact s3 this time for those clouddb hosts (and sanitarium)
[07:21:17] <federico3>	 @marostegui see the conversation in the backlog: the sanitization script was running with --section s3 but clouddb1013 has no s3 section and the script crashes out on it. I think the script needs a tiny change to detect missing sections on clouddb hosts and skip them
[07:22:00] <marostegui>	 federico3: Sure, but why are all the other hosts alerting? :)
[07:22:36] <marostegui>	 federico3: clouddb1013 does have s3
[07:23:48] <federico3>	 only 2 hosts are alerting now, I suspect the script did not reach them because it crashed on clouddb1029
[07:24:03] <marostegui>	 I got emails for 5 hosts
[07:25:18] <marostegui>	 Also, where did you see that clouddb1013 has no s3? I can see it and connect to it
[07:26:52] <federico3>	 when did you get other emails? I'm seeing only 2 from this morning
[07:28:05] <federico3>	 the script crashes on clouddb1029 because it has no s3 (not clouddb1013) so it did not continue and left clouddb1013 not sanitized
[07:28:16] <marostegui>	 I got all the emails today 
[07:28:44] <federico3>	 I'm seeing only 2 
[07:29:02] <marostegui>	 I guess you need to review your filters :)
[07:29:23] <federico3>	 https://www.irccloud.com/pastebin/KBGnKc2g/
[07:29:55] <marostegui>	 federico3: db1154 isn't one of the hosts that got the email
[07:29:58] <marostegui>	 it is all the hosts below them
[07:30:24] <federico3>	 yesterday I received the email from db1154 as well and it had data and it was sanitized successfully
[07:31:20] <marostegui>	 federico3: Again, that is not one of the hosts that has emailed us, yesterday we got emails for db1154 and all the hosts below it. What I am saying is that I still got the emails for all the hosts below it
[07:31:37] <federico3>	 I also received the email for clouddb1017 only on yesterday
[07:32:22] <marostegui>	 Anyway, I've fixed this everywhere
[07:34:08] <marostegui>	 federico3: As I said, you probably have to check your filters 
[07:34:12] <marostegui>	 Private data found at clouddb1013
[07:34:12] <marostegui>	 Inbox
[07:34:12] <marostegui>	 root via wikimedia.org 
[07:34:12] <marostegui>	 7:09 AM (2 hours ago)
[07:34:12] <marostegui>	 to cwilliams, fceratto, jcrespo, me, root
[07:35:07] <marostegui>	 In any case, I've fixed all of this
[07:37:52] <federico3>	 to clarify: yesterday I received the emails for db1154, an-redacteddb1001, clouddb1013, clouddb1017.   I ran the script twice but it crashed; today I received the emails only from an-redacteddb1001 and clouddb1013 and the logs showed that the data was gone from db1154 
[07:39:31] <marostegui>	 Yes, as mentioned above, the problem wasn't on db1154 but on the rest
[07:39:33] <federico3>	 ah, found them
[07:39:34] <marostegui>	 I've fixed it
[07:40:02] <federico3>	 gmail flagged them as spam, nice
[07:40:19] <marostegui>	 classic gmail prioritization!
[07:40:35] <cezmunsta>	 That has happened with other emails, such as those to noc@ iirc
[07:44:26] * Emperor mutters something unparliamentary about gmail
[07:47:49] * cezmunsta wonders why my IRC connection is to LA
[07:58:16] <jynus>	 Would anyone want a 10 minute handover meeting? I don't have anything urgent, just want to give some pointers of misbehaving hw and so, specially for the dbas
[07:59:26] <marostegui>	 jynus: Maybe it is easier to send it via email so it is written and can be checked anytime?
[07:59:51] <marostegui>	 I know it is probably more tedious but may be easier for us to reference it back when needed
[08:00:13] <jynus>	 ok, to whom?
[08:00:23] <marostegui>	 I'd say everyone?
[08:00:56] <jynus>	 ok
[08:01:03] <marostegui>	 thank you!
[08:30:39] <jynus>	 I will try to do a reboot speedrun, mistakes may be made potentially (as in in alerts going off) to try to do it before friday, I apologize in advance
[08:30:54] <marostegui>	 thanks jynus 
[08:47:03] <cezmunsta>	 Doing s6 codfw switchover
[10:25:15] <jynus>	 there is something weird going on, but I don't have time to investigate
[10:25:39] <jynus>	 if I start mariadb on a multiinsance host shortly after reboot, it sets its socket on /run/mysql.sock
[10:25:57] <jynus>	 instead of the configured one at /run/mysqld.SECTION.sock
[10:26:43] <jynus>	 shortly after, it gets fixed without doing anything
[10:26:45] <cezmunsta>	 Do you have a specific example?
[10:27:01] <jynus>	 It just happened to me on db2239
[10:27:08] <jynus>	 and on another one, let me search
[10:28:09] <jynus>	 I notice because I get "[ERROR] Another process is already listening on the socket file '/run/mysqld/mysqld.sock'"
[10:28:31] <jynus>	 but that's not the one on config
[10:30:50] <cezmunsta>	 What did you execute to up to the point where you started MariaDB?
[10:31:00] <jynus>	 before? nothing
[10:31:13] <jynus>	 my first command was systemctl start mariadb@section
[10:31:32] <cezmunsta>	 So, you the reboot etc all manually?
[10:31:34] <jynus>	 and it didn't use the right socket
[10:31:47] <jynus>	 Me the reboot?
[10:32:06] <cezmunsta>	 db2239 uptime = 7 mins
[10:32:34] <jynus>	 yes, that was the first command after the reboot
[10:32:48] <jynus>	 obviously not the first command the server has ever received
[10:32:50] <marostegui>	 jynus: that's because puppet didn't run
[10:33:04] <jynus>	 but the config should be there (?)
[10:33:30] <jynus>	 as in, it was not a reimage, just a simple reboot
[10:34:16] <jynus>	 could it be due to an upgrade?
[10:34:35] <jynus>	 a package upgrade?
[10:34:45] <jynus>	 that overrides puppet and it requires a new puppet run?
[10:37:55] <cezmunsta>	 2026-08-13T10:24:19.973747+00:00 db2239 puppet-agent[1942]: (/Stage[main]/Mariadb::Config/File[/etc/mysql/my.cnf]/ensure) removed (corrective)
[10:39:09] <cezmunsta>	 That matches the error
[10:39:20] <cezmunsta>	 Aug 13 10:24:09 db2239 mysqld[2201]: 2026-08-13 10:24:09 0 [ERROR] Another process is already listening on the socket file '/run/mysqld/mysqld.sock'. Aborting
[10:41:13] <jynus>	 probably created by a new mariadb package installation, right?
[10:41:14] <cezmunsta>	 Perhaps we should not be removing the file and instead ensure it has content so that it doesn't get changed?
[10:42:29] <jynus>	 up to you, I got my answer (run puppet after reboot), I will keep rebooting
[13:23:00] <marostegui>	 federico3: we are going to have to rename db-test hosts, dbctl isn't liking db- regex and I don't want to keep adding regex for them, can we rename/reprovision. I am going to add a comment on the task so we can follow up there.
[13:24:30] <federico3>	 the dash required tweaks in other places in puppet, scripts, cookbooks etc so maybe having dbXXXX would be good in general
[13:26:37] <marostegui>	 federico3: https://phabricator.wikimedia.org/T427059#12212193
[15:35:11] <cezmunsta>	 Doing s7 codfw switchover