[03:49:44] <jinxer-wm>	 FIRING: SystemdUnitFailed: swift_rclone_sync.service on ms-be1069:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[07:33:54] <jynus>	 bacula daily backups finished its batch, dumps are soon to finish probably  but still ongoing re: m1
[07:36:43] <jynus>	 s5 dumps for codfw failed, it normally takes 30 minutes to run, can I rerun it now or should I wait?
[07:44:16] <jynus>	 I am reruning them, I had some weird permission error, but we lose nothing if it doesn't complete
[07:49:44] <jinxer-wm>	 FIRING: SystemdUnitFailed: swift_rclone_sync.service on ms-be1069:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[07:53:56] <jinxer-wm>	 RESOLVED: SystemdUnitFailed: swift_rclone_sync.service on ms-be1069:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[08:07:45] <marostegui>	 jynus: thanks, so should I wait a bit for m1 master switch?
[08:07:56] <marostegui>	 jynus: also, do we store dbprov content in bacula?
[08:10:31] <marostegui>	 Nevermind, I just saw the list of dbprov in bacula yep
[08:20:30] <jynus>	 yes
[08:20:38] <jynus>	 all backups finished correctly
[08:20:49] <marostegui>	 sweet
[08:20:55] <marostegui>	 I will get to it after switching ms1
[08:21:33] <jynus>	 I may go now to take a coffee break, feel free to merge my patch when done, no blockers from me
[08:22:35] <marostegui>	 ok thanks
[08:31:42] <marostegui>	 m1 switch done
[08:41:10] <marostegui>	 Going to start with ms1 now
[09:40:23] <jynus>	 was the revert created just in case or something happened?
[09:40:55] <marostegui>	 no, Reason for revert: Old master reimaged, switching back to it.
[09:40:59] <jynus>	 ah, I see it was temporary only
[09:41:14] <jynus>	 I was worried something went wrong, all good
[09:41:45] <jynus>	 if it is going to be done today, it will be great until 20h
[09:42:42] <jynus>	 I was confused as the reason wasn't part of the initial notification email
[09:43:48] <marostegui>	 yes, I will do it in a bit, the reimage is finishing
[09:45:28] <jynus>	 no rush, and can be done another day, no issues if needed
[09:45:50] <jynus>	 thank you a lot for keeping me updated, marostegui <3
[10:23:04] <marostegui>	 federico3: dbctl needs your changes to be committed for db1177, please also see gerrit
[10:24:50] <federico3>	 oh wow, why did it not commit that... commiting now
[10:25:02] <marostegui>	 it is never committed as far as I know
[10:25:11] <marostegui>	 You sent the puppet patch but never committed in dbctl
[10:27:05] <federico3>	 I'll update the cookbook
[10:27:39] <marostegui>	 The cookbook doesn't remove the host from dbctl, that needs a puppet patch
[10:29:43] <marostegui>	 https://phabricator.wikimedia.org/T433474#12201720
[10:29:46] <marostegui>	 federico3: ^
[10:30:01] <federico3>	 yes I created it
[10:30:36] <marostegui>	 federico3: created what? I am linking just the comment that was missing on the task (which is what the cookbook states to do btw)
[10:32:03] <federico3>	 I mean I created the puppet patch and I'm adding automation
[10:45:19] <marostegui>	 federico3: yeah, but did you see the comment above?
[10:47:54] <federico3>	 @marostegui I'm not sure what are you pointing to with https://phabricator.wikimedia.org/T433474#12201720 ?  it links to "This is ready for DCOps."
[10:49:28] <marostegui>	 federico3: yes, that was missing and that what I was pointing you to. The decommission coookbook also states that needs to be done
[11:33:15] <cezmunsta>	 Doing s8 codfw switchover
[12:47:11] <marostegui>	 jynus: db2201 with replication stopped for 2 days, almost 3, is that known/expected?
[12:47:18] <marostegui>	 backup source for x1
[12:50:06] <cezmunsta>	 marostegui: in terms of the cloning cookbook, when picking the source which is the normal choice? the backup source for the section, the host that is being replaced (if applicable), the candidate master or a "random" choice?
[12:50:22] <marostegui>	 cezmunsta: Normally I pick the one being replaced yeah
[12:50:26] <marostegui>	 If not any other
[12:50:41] <cezmunsta>	 +1
[12:51:32] <jynus>	 weird
[12:51:41] <jynus>	 checking
[12:54:11] <jynus>	 it seems mysql started 12h ago, but I didn't touch that host since a long time
[12:54:39] <jynus>	 and there was no log in since Jul 10
[12:54:54] <marostegui>	 what's the uptime?
[12:54:59] <marostegui>	 of the host? 
[12:55:01] <jynus>	 on the host?
[12:55:07] <jynus>	 32 day
[12:55:14] <marostegui>	 weird then :-/
[12:55:17] <jynus>	 it is not a crash because otherwise it had not started
[12:55:40] <jynus>	 checking the logs to see at least what happened
[12:55:58] <jynus>	 yeah, there was a mysql crash
[12:56:14] <jynus>	 Aug 11 00:02:03 db2201 mysqld[6569]: 260811  0:02:03 [ERROR] /opt/wmf-mariadb1011/bin/mysqld got signal 7 ;
[12:56:19] <jynus>	 it happened tonight
[12:56:25] <jynus>	 but weird it happened on both instances
[12:57:35] <jynus>	 actually the one on x1 happened on the Aug 08 17:11:30 db2201 mysqld[6648]: 260808 17:11:30 [ERROR] /opt/wmf-mariadb1011/bin/mysqld got signal 7 ;
[12:57:56] <jynus>	 still, it is weird it happened within 2 days of each other on the same host
[12:58:03] <jynus>	 it could be storage-related
[12:58:33] <jynus>	 I will create a task
[12:59:00] <jynus>	 maybe start replication as is , but I will rebuild the data hosts 
[13:00:13] <cezmunsta>	 jynus: could it be memory related?
[13:00:19] <jynus>	 yeah, that too
[13:00:28] <jynus>	 I will check hw logs
[13:00:31] <jynus>	 in general
[13:00:34] <jynus>	 for anything weird
[13:03:51] <jynus>	 I've pasted both full logs if curious: https://phabricator.wikimedia.org/T434532
[13:04:23] <cezmunsta>	 Whilst reported for a newer version (and incomplete)... https://jira.mariadb.org/browse/MDEV-34197 
[13:04:27] <marostegui>	 SELECT /*!40001 SQL_NO_CACHE */ * FROM `cebwiki`.`templatelinks` WHERE (`tl_from` >= 38177>
[13:04:33] <marostegui>	 isn't this table huge?
[13:04:39] <marostegui>	 sounds familiar for cebwiki
[13:05:10] <jynus>	 one actionable for dba automation, is tat I would love to reenable replication control with fixed minimum for backup sources
[13:05:43] <jynus>	 right now it alerts when lag > 0, so I have to disable it for backup sources because I regularly stop replication for backup taking
[13:06:19] <jynus>	 it would be nice to have a lag parameter that is like 1 day so it alerts if it gets delayed beyond that
[13:09:12] <jynus>	 cezmunsta: that's unrelated, mysql is full of self asserts and it crashes itself to prevent extra corruption (I am guessing you searched the signal message)
[13:09:55] <jynus>	 I can ran SHOW GLOBAL STATUS with no crash
[13:09:58] <jynus>	 *run
[13:15:02] <cezmunsta>	 jynus: It was more that there was a recent enough MariaDB with a sigbus ... yet Oracle ones all rather old
[13:15:57] <jynus>	 sorry, one ask: maybe I am doing something wrong with new authentication or something, but I cannot access ssh or http out of bad management for this host
[13:16:03] <jynus>	 can someone try?
[13:16:12] <jynus>	 db2201.mgmt.eqiad.wmnet
[13:18:12] <cezmunsta>	 jynus: I get to "(cwilliams@db2201.mgmt.codfw.wmnet) Password:" ... have you got anything extra in your ssh-config?
[13:18:52] <cezmunsta>	 :q
[13:19:04] <jynus>	 no, but you need to proxy through a cumin to access the mgmt ssh or http
[13:19:26] <cezmunsta>	 I am using wmf-laptop and that has the mgmt nodes in
[13:19:32] <jynus>	 ah, ok
[13:21:18] <marostegui>	 jynus: works for me, are you using the new wmfroot user?
[13:21:48] <jynus>	 ok, if it works for you then I must have something wrong on my side, could be that
[13:22:29] <marostegui>	 https://phabricator.wikimedia.org/T434532#12202633
[13:24:10] <jynus>	 oh, no it is me being silly
[13:24:19] <jynus>	 trying to connect to db2201.eqiad.wmnet
[13:25:31] <jynus>	 I was like, "this worked yesterday", silly me
[13:26:38] <jynus>	 cezmunsta was right, it is memory
[13:30:12] <jynus>	 I've send it to dc ops and hopefully it is more stable before my vacations
[13:30:32] <marostegui>	 jynus: do we sent snapshots of say, s1 to bacula?
[13:30:38] <marostegui>	 snapshots, not logical backups
[13:30:48] <jynus>	 not snapshots at the moment, only dumps
[13:30:58] <marostegui>	 got it, that's what I suspsected, thanks
[13:31:20] <jynus>	 we could send snapshots but we don't have have space atm, it is prepared for that but I had to use spare capacity to handle mediabackups
[13:31:30] <marostegui>	 yeah, no problem
[13:31:32] <marostegui>	 thjanks
[13:31:39] <jynus>	 but it is ready for it if it was required + capacity
[13:31:58] <jynus>	 we have a week of buffer however, which is what I will use for this
[13:32:17] <marostegui>	 so snapshosts are sent cross dc
[13:32:22] <marostegui>	 "as an offsite"
[13:32:43] <jynus>	 not technically, but they are generated redundantly on both
[13:32:48] <marostegui>	 ah ok!
[13:32:57] <jynus>	 so we generate all backups on both sites
[13:33:23] <jynus>	 but cross send to backula only dumps atm, snapshots not but it would be trivial to activate if we had the capacity
[13:33:43] <jynus>	 we use bacula as the cross-dc slow storage for longer term
[13:34:02] <jynus>	 and for that we only priotitize dumps, which is usually what is used first long term
[13:34:12] <marostegui>	 yeah, that I knew, thanks
[13:34:26] <jynus>	 but snapshots could be done, but it is quite expensive, it is like 20 GB a day
[13:34:33] <jynus>	 sorry
[13:34:38] <jynus>	 20 TB, he he
[13:35:28] <jynus>	 Pampinus is a good place to kinda get the big picture of where things are generated
[13:37:14] <jynus>	 let me see if I have a replacement for db2201 instances fastly
[13:37:58] <marostegui>	 jynus: what's the url for pampinus?
[13:38:01] <jynus>	 I would suggest not to use last snapshot for s5 in codfw, I will refresh it asap from anotehr host
[13:38:57] <jynus>	 marostegui: https://wikitech.wikimedia.org/wiki/MariaDB/Backups#Dashboard
[13:39:26] <jynus>	 so in this case, we still have the eqiad backups untouched
[13:39:29] <marostegui>	 Ah thank you, I looked for Pampinus on wikitech and only found the repo link
[13:39:38] <jynus>	 I will add the name there
[13:39:44] <marostegui>	 I was about to do it
[13:39:48] <marostegui>	 go ahead
[13:41:17] <jynus>	 it will eventually get to kubernetes, one day(TM)
[13:43:06] <tappof>	 Hi there! Related to T434502 (and therefore to T379942), we'd like to roll out https://gerrit.wikimedia.org/r/c/operations/puppet/+/1324302. Emperor, Amir1, could you please take a look and let us know if it's okay to proceed? Thanks!
[13:43:07] <stashbot>	 T434502: Swift logging is swamping centrallog hosts - https://phabricator.wikimedia.org/T434502
[13:43:07] <stashbot>	 T379942: Gradually drop all thumbnails as a one-off clean up - https://phabricator.wikimedia.org/T379942
[13:45:25] <jynus>	 Re: db2201 x1 has redundancy, s5 not, which is great because that's the most recent crash
[13:45:41] <Emperor>	 tappof: +1
[13:46:00] <tappof>	 thx Emperor 
[13:48:42] <jynus>	 Re: db2201 the intention is to recover the oldest snapshot we have for s5, from 2026-08-05, of which I have a copy after servicing
[14:59:45] <cezmunsta>	 Doing s2 codfw switchover