[01:25:25] <jinxer-wm>	 FIRING: SystemdUnitFailed: systemd-journal-flush.service on ms-be2075:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[03:50:25] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: systemd-timedated.service on ms-be2051:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[05:24:48] <jinxer-wm>	 FIRING: PuppetFailure: Puppet has failed on ms-be2051:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[05:51:14] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on s4 on db2219 is CRITICAL: 216.5 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2219&var-port=9104
[05:59:14] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on s4 on db2219 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2219&var-port=9104
[07:50:25] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: systemd-timedated.service on ms-be2051:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[08:17:51] <Emperor>	 that's going to be more dead hardware
[08:55:08] <marostegui>	 I have so many things to do that I created a task yesterday and I don't know why and what for XD
[09:02:49] <volans>	 create a task today to review existing tasks :-P
[09:08:06] <Emperor>	 and assign it to v.olans ;p
[09:24:48] <jinxer-wm>	 FIRING: PuppetFailure: Puppet has failed on ms-be2051:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[09:27:38] <marostegui>	 volans: XDDD
[09:28:00] <marostegui>	 Seriously I have no idea why I created it
[10:30:34] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on s7 on db2221 is CRITICAL: 58.2 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2221&var-port=9104
[10:31:34] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on s7 on db2221 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2221&var-port=9104
[10:42:29] <marostegui>	 I need to change x1 to be SBR for a bit
[10:42:31] <marostegui>	 Amir1: ^
[10:42:52] <Amir1>	 Sure@
[10:43:15] <Amir1>	 I have nothing on x1
[10:43:20] <marostegui>	 you do
[10:43:22] <marostegui>	 https://phabricator.wikimedia.org/T385645
[10:43:23] <marostegui>	 XD
[10:51:11] <marostegui>	 I am going to hjave to reclone the broken x1 host, damn RBR
[10:51:54] <dhinus>	 marostegui: I was on PTO yesterday, I can do 1017 today
[10:52:28] <dhinus>	 btw I see there is some lag on clouddb1013 that I upgraded on Friday, not sure what caused it
[10:52:41] <jynus>	 marostegui: want me to recover it from backups?
[10:53:20] <marostegui>	 dhinus: Probbably schema changes or indexs rebuilds
[10:53:23] <marostegui>	 jynus: no, thanks it is fine
[10:53:26] <Amir1>	 > s7 	eqiad 	snapshot 	fresh 	5 hours ago 	916.9 GB 	-2.7 % 	— 
[10:53:48] <Amir1>	 marostegui: I mean, I'm not running it :P
[10:56:05] <Amir1>	 the s2 wikireplicas are lagging because the sanitarium master is being rebuilt so that's fine, I don't know which deamon of clouddb1013 is lagging
[10:56:29] <dhinus>	 replag on clouddb1013 is going down so I will ignore it for now
[10:57:03] <dhinus>	 something is wrong on db1156 (s2) too https://orchestrator.wikimedia.org/web/cluster/alias/s2
[10:58:55] <marostegui>	 dhinus: That is index being rebuilt as Amir1 said
[10:59:06] <marostegui>	 [11:53:20]  <marostegui> dhinus: Probbably schema changes or indexs rebuilds
[10:59:07] <marostegui>	 And that ^
[10:59:17] <marostegui>	 We are rebuilding many hosts/indexes
[10:59:33] <dhinus>	 ok thanks :)
[10:59:52] <dhinus>	 I will upgrade+reboot clouddb1017, and the other clouddbs after
[11:00:19] <marostegui>	 great thanks
[11:00:25] <dhinus>	 s2
[11:00:50] <Amir1>	 I want to run a schema change on s2 for a half a week now but you're just rebuilding everything 😡
[11:01:06] <dhinus>	 "s2": sorry that was a search string I didn't mean to post :P
[11:01:07] <marostegui>	 Amir1: I am trying to avoid more crashes
[11:01:36] <Amir1>	 I know :P I'm just grumpy 
[11:02:49] <dhinus>	 Amir1: "which daemon of clouddb1013" < s1 is lagging there, but getting better
[11:03:11] <marostegui>	 dhinus: that comes from index rebuilds
[11:03:29] <dhinus>	 all good
[11:04:07] <Reedy>	 marostegui:  I think you might want to stop T385645
[11:04:07] <stashbot>	 T385645: Drop event_variant column from echo_event - https://phabricator.wikimedia.org/T385645
[11:04:15] <marostegui>	 Reedy: It is stopped
[11:05:06] <marostegui>	 It was only done in a host, which broke and it's been depooled for a while now
[11:05:37] <Reedy>	 was still erroring ~20 mins ago
[11:06:07] <marostegui>	 Ah yes, the stupid script was still running
[11:06:08] <marostegui>	 My bad
[11:06:11] <marostegui>	 Depooling and killing it
[11:06:14] <marostegui>	 Thanks for the heads up
[11:06:21] <Reedy>	 heh, np :)
[11:06:39] <marostegui>	 I should really not work on other stuff when I am on clinic duty
[12:50:04] <marostegui>	 Amir1: https://phabricator.wikimedia.org/T385645#10525207
[12:50:47] <Amir1>	 ah
[12:51:07] <Amir1>	 damn. I think it's NULL but explicitly? 
[12:51:36] <Amir1>	 yup
[12:51:49] <Amir1>	 let me backport the patch to remove that
[12:51:54] <marostegui>	 Yeah
[12:51:58] <marostegui>	 I am going to revert this on the host
[12:52:01] <marostegui>	 aka: reclone
[12:52:03] <marostegui>	 (again)
[12:52:39] <Amir1>	 sorry :(
[12:52:50] <marostegui>	 It is fine, that's why we run them on replicas first 
[12:52:53] <marostegui>	 To catch these things
[13:10:35] <Emperor>	 Can I get +1s for 2 linked changes, please, to prep the old and now drained ms-be2* nodes for decom? https://gerrit.wikimedia.org/r/c/operations/puppet/+/1117535 and https://gerrit.wikimedia.org/r/c/operations/puppet/+/1117536
[13:18:01] <Emperor>	 Thanks :)
[13:21:03] <jynus>	 Amir1: not needing a full review, just an ok to proceed with https://gerrit.wikimedia.org/r/c/operations/puppet/+/1116846/4
[13:22:25] <Amir1>	 +1'ed!
[13:22:28] <jynus>	 thanks
[13:22:43] <jynus>	 will deploy the removals on m1 primary
[14:35:29] <Amir1>	 FYI, I'm running the file migration on s5, basically only dewiki has a lot of files (100K)
[14:35:38] <Amir1>	 you might see a lot of writes there
[14:53:11] <swfrench-wmf>	 hi folks - any objections / concerns if I release conftool soon to enable the min pooled parsercache sections safety check? (T383324)
[14:53:11] <stashbot>	 T383324: Prevent too many parsercache sections from being depooled - https://phabricator.wikimedia.org/T383324
[14:54:06] <Amir1>	 none on my side
[14:54:14] <Amir1>	 only thank you for picking it up!
[14:54:39] <swfrench-wmf>	 no problem, and thanks for your patience with getting it live taking a while :)
[15:50:15] <swfrench-wmf>	 all done. verified with `dbctl config generate` and `dbctl config diff` which, despite being non-mutating operations, exercise the same checks as a `commit` would
[15:50:33] <swfrench-wmf>	 I'll update the wikitech docs about depooling parsercache sections in a bit
[18:42:38] <mutante>	 regarding Bacula backups: I seem to remember there was something to mark a backup::set as "keep around forever", as in ..archive it and never rotate it away. Is that right? What was it? Or would that be default behaviour if I have an existing regular backup and then just remove it from puppet.
[18:43:59] <mutante>	 what I want is "one last full backup, keep it forever and stop make new ones"