[08:15:49] Hi folks, could I get a +1 on https://gerrit.wikimedia.org/r/c/operations/puppet/+/1266138 please? we have two rows worth of eqiad systems to reimage, this drains the first 3 swift backends. [08:20:57] done [08:21:49] TY [08:45:37] reciprocation? https://gerrit.wikimedia.org/r/c/operations/puppet/+/1266148 [08:50:12] +1 [08:55:02] thank you [10:48:52] o/, marostegui or Amir1: I pinged on task https://phabricator.wikimedia.org/T419980 about the table swap for the new process for ICU upgrade and didn't get a response, could we please schedule a time to do it? [10:52:42] Hi Raine - I came back from 3 weeks out, so I am still catching up [10:52:46] I will take a look [10:52:49] thank you <3 [10:53:42] if you'd like to have more time before doing the procedure, feel free to suggest a later date, I'd just like to fix one, so CommRel have time to announce it [11:13:42] Raine: The dates you've proposed will not work, as we have public holidays till tue [11:14:00] And I think we need to discuss the fact that it cannot be done gradually but all or nothing [11:14:02] I will comment on the task [11:36:56] ack, thank you marostegui <3 [11:41:48] FIRING: PuppetFailure: Puppet has failed on ms-be1065:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [11:48:17] 4🦊🐟, I've filed one bust-disk ticket today already [12:09:38] T422011 filed for ms-be1065, alert silenced [12:09:38] T422011: Disk (sdt) failed in ms-be1065 - https://phabricator.wikimedia.org/T422011 [12:49:05] I added some basic docs to https://wikitech.wikimedia.org/wiki/MariaDB#HAProxy [12:57:07] federico3: One clarification, only if the primary goes down it needs restarting, if the backup ones does, and then it comes back, haproxy will pick it up [13:06:32] I updated the page to make it more clear [13:06:57] thank you! [14:01:01] The reason to not switch back is the lack of stonith, and previous cases where it flapped up and down (like phabricator in the old days with myisam) [14:01:57] at some point that architecture should be revised, it was never changed when I onboarded due to vip constraints [14:02:35] vip as in floating virtual ip constraints [18:09:45] Another day, another inefficient querying in mw :D https://phabricator.wikimedia.org/T422069