[10:07:09] Krinkle: thank you for the heads up re: pontoon video, I've used the opportunity to re-record the demo with asciinema and updated wikitech, please LMK if that works [10:07:39] I'd like to update the plain text file to wikitech but that's not allowed by default at least [10:52:11] qq for anyone please. Wikitech says I should use mwmaint1002 for my task. motd says quite clearly 'Do not use this server'. I guess I should use mwmaint2002 but I thought I'd check here first. I'm enabling U2F for my own account. https://wikitech.wikimedia.org/wiki/CAS-SSO#Enabling_U2F_for_an_account [10:54:12] btullis: if all else fails, try running it on mwmain2002 and see what happens :) [10:57:59] btullis: the MOTD is only for people running Mediawiki maintenance scripts [10:58:22] running LDAP related scripts/commands can be done from both servers independent of the current active datacentre [10:59:20] OK, thanks both. So the `modify-mfa` is fine to run on either of the two mwmaintX002 servers at any time, then? If so, I'll update the wiki. [11:02:02] ack, thanks for updating [11:06:22] Done. Thanks. [14:19:51] https://codeascraft.com/2021/06/15/improving-the-deployment-experience-of-a-ten-year-old-application/ [14:28:14] hello! I have a couple puppet patch pending for review and could use a pair to push them please :] [14:28:14] the first drops a Hiera yaml file that is not doing any effect, it is a mistake I made in an earlier patch and the file should just be deleted ( https://gerrit.wikimedia.org/r/c/operations/puppet/+/673286 ) [14:28:14] the second is to add a motd for all deployment-prep / beta cluster instances to mention the WMCS terms. That would fix a task from 2015! :] https://gerrit.wikimedia.org/r/c/operations/puppet/+/699207 [14:28:54] we have an incident right now, but I can look anyways [14:29:36] oh hi mutante :] [14:30:08] one is a noop, the other is already cherry picked on beta. So should be straightforward [14:33:52] hashar: all done! if you have those feel free to just add myself and others in gerrit. we will be notified [14:34:23] mutante: <3 !! [14:35:16] and that fixes a task filed back in 2015 :] [14:36:03] nice! [14:36:43] yeah I had the crazy idea to revisit the task I have filed ordered by creation date [14:36:47] and try to fix the oldest ones [14:37:02] then went to write a short post about my experiment https://phabricator.wikimedia.org/phame/post/view/244/shrinking_the_tasks_backlog/ :D [14:37:06] :) that brings memories. hehe [14:39:21] definitely [14:39:32] and some are surprising super easy to fix [14:39:50] anyway thx for the merge, I will let you deal with the ongoing incident [14:43:46] it's looking good. everything recovering right now. ongoing work on a switch [15:26:56] If there is other sre that, like me, cannot be very useful right now (not actively changing stuff) and is around, I would request a co-IC please, specially someone from americas tz more fresh [15:27:14] I will be around, but I am getting a bit overloaded with all updates on several channels [15:31:28] jynus: update from us if you didn't see, remote-hands in codfw power-cycled the switch, but no output on console port so it's confirmed as dead. [15:31:52] thanks, topranks, as I said above I am losing track of somethings, so much appreciated [15:32:13] We will confirm if we decide otherwise, but consensus right now is asking remote-hands to do full switch replace/recable is more risky than running without that switch over the weekend. [15:32:33] So likely we will leave for Monday when Papaul is back.