[08:33:25] <godog>	 I'll resolve the stray incident open from yesterday's incidents
[08:34:34] <_joe_>	 uhm I think it might fire again soon :/
[08:36:24] <elukey>	 hello folks
[08:36:35] <elukey>	 if nobody opposes I'd merge/build/rollout wmf-certificates for https://gerrit.wikimedia.org/r/c/operations/debs/wmf-certificates/+/742485
[08:36:51] <elukey>	 (rollout via debdeploy)
[08:45:50] <_joe_>	 elukey: lemme take a quick look :)
[08:47:48] <_joe_>	 lgtm
[08:57:23] <elukey>	 thanks :)
[13:24:25] <arturo>	 elukey: you were about to merge sslcert::trusted_ca: ensure cert bundle readability for group/others (08a534c57d) ?
[13:24:37] <elukey>	 arturo: yes I was about to ask, thanks :)
[13:24:45] <arturo>	 cool
[14:52:11] <cdanis>	 godog: any idea why we get stray lingering VO incidents so often?
[14:55:30] <godog>	 cdanis: yes and no, happens only for host alerts so it has sth to do with the formatting/parsing I think, T264016 is what I'm using
[14:55:31] <stashbot>	 T264016: Host page did not auto-resolve in VO - https://phabricator.wikimedia.org/T264016
[15:05:53] <majavah>	 could someone have a look at <+icinga-wm> PROBLEM - Keyholder SSH agent on deploy1002 is CRITICAL: CRITICAL: Keyholder is not armed. Run keyholder arm to arm it. https://wikitech.wikimedia.org/wiki/Keyholder please?
[15:06:27] <jbond>	 majavah: hmm thats probably related to me removinf deployment-puppetboared will take a look
[15:43:23] <moritzm>	 jbond, herron: shall we take another stab at the mx2001/LDAP change?
[15:43:46] <jbond>	 moritzm: good with me
[15:43:47] <herron>	 moritzm: sure
[15:44:28] <jbond>	 ack let me get the cr up
[15:44:37] <moritzm>	 ack
[15:45:30] <jbond>	 ack merging now
[15:46:08] * jbond applying
[15:47:18] <jbond>	 applied and `sudo exim  -bt   otrs-test@wikimedia.org ` looks goot to me
[15:48:27] <moritzm>	 test mail via mx2001 also arrived fine
[15:49:03] <moritzm>	 just sent a test mail to otrs-test@w.o from external
[15:49:24] <moritzm>	 ^ akosiaris: can you please check if you can see a new ticket in the test queue?
[15:49:45] * akosiaris looking
[15:49:55] * jbond also recived mail to jbond@w.o
[15:50:45] <akosiaris>	 I see 2
[15:51:18] <jbond>	 i sent one from the cli with subject: test-mx2001
[15:51:42] <akosiaris>	 one from jbond@mx2001, 3m ago and one from jmm@viruvalge 2m ago
[15:51:50] <moritzm>	 excellent, thanks
[15:51:52] <jbond>	 ahh cool thanks 
[15:52:12] <herron>	 looking good, paniclog empty as well
[15:52:27] <jbond>	 yes and i dont see anything bad in the exim log
[15:52:35] <jbond>	 should we leave it 245 huors then do mx1001?
[15:52:52] <moritzm>	 jbond, herron: I'd say we keep this in the current config until tomorrow, then I'll revert the patches that made routing prefer mx1001 for a day
[15:52:55] <moritzm>	 and then mx1001?
[15:53:20] <moritzm>	 but can also do it quicker, fine either way
[15:53:37] <herron>	 sgtm, +1 for the cautious approach
[15:53:44] <jbond>	 moritzm: i thinkyou proposal sgtm 
[15:53:51] <moritzm>	 ok
[16:19:36] <kormat>	 "In this way our code should run and be testable on Pontoon and Deployment-prep without too many swearing or moments of internal sadness."
[16:19:43] <kormat>	 elukey: 💜 💜 💜
[16:22:53] <RhinosF1>	 elukey: can I please stick that on bash?
[16:23:26] <elukey>	 :)
[16:23:51] <elukey>	 probably too much swearing, just realized it :D
[16:24:39] <RhinosF1>	 !bash <elukey> In this way our code should run and be testable on Pontoon and Deployment-prep without too many swearing or moments of internal sadness.
[16:24:39] <stashbot>	 RhinosF1: Stored quip at https://bash.toolforge.org/quip/hWGpcX0B8Fs0LHO5l3ng
[16:24:48] <RhinosF1>	 elukey: it still sounds fun
[17:23:50] <legoktm>	 majavah: were there any issues from switching deployment-deploy03?
[17:24:56] <majavah>	 legoktm: I don't think there were any major issues, it went suprisingly well
[17:25:25] <apergos>	 when did that happen btw?
[17:25:36] <majavah>	 yesterday evening my time
[17:26:02] <apergos>	 uh
[17:26:06] <apergos>	 which time is your time? :-)
[17:26:27] <majavah>	 EET = UTC+2
[17:26:42] <apergos>	 ah, literally my time!  
[17:27:03] <majavah>	 that's what I said
[17:27:07] <apergos>	 so probably after I was on it then.  so I can't verify that it's all good. maybe next time
[17:27:16] <apergos>	 "next" time, see what I did there <--
[17:27:41] <legoktm>	 awesome :D
[17:27:43] <apergos>	 uh what I mean is that your time is my time :-P
[17:27:55] <apergos>	 maybe I need a time out...
[21:09:10] <tltaylor>	 hi folks. Comms is asking for attention on T296570 in support of a microsite launch they intend to start tomorrow 
[21:09:10] <stashbot>	 T296570: Setup subdomain for Foundation messaging site - https://phabricator.wikimedia.org/T296570
[21:13:26] <_joe_>	 herron: ^^
[21:15:21] <sukhe>	 I am submitting a patch for this
[21:15:46] <herron>	 thanks sukhe I was just looking into the same
[21:16:03] <mutante>	 can't believe it's "launch tomorrow" once again
[21:16:30] <_joe_>	 mutante: it's not, the task is open since a few days.
[21:17:01] <_joe_>	 and given it's a one-line patch, the somewhat short timing isn't really an issue.
[21:22:09] <Krinkle>	 Are we going to find privacy, security, performance, design, a11y, and closed-source issues post-launch of the same nature we found the last five times we launched a WP-VIP site?
[21:22:15] <herron>	 TLS does not work for this site when testing via /etc/hosts
[21:24:28] <tltaylor>	 who is responsible for managing the VIP hosted sites?
[21:30:08] <_joe_>	 Krinkle: at the very least, this is a private site, meaning it's impacting employees only and so it has a limited scope and I think we can do that yes :)
[21:30:25] <Krinkle>	 I believe in recent years, since the redesign of the foundation website, most of them are managed by Comms and outsourced, no longer involving  the engineering departments (not in a planned way at least, we do generally end up helping post-launch ad-hoc).
[21:30:28] <_joe_>	 tltaylor: what do you mean with "responsible for managing"? I guess it's automattic
[21:31:05] <legoktm>	 I would assume having a proper TLS cert was blocked on the DNS being enabled
[21:31:16] <tltaylor>	 I'm trying to understand who would have the ability to address the issues Timo raised
[21:31:18] <_joe_>	 tltaylor: we can only manage the DNS record, but as herron pointed out, the certificates will need to be installed by who manages the infrastrcuture serving the sites, which is not us
[21:31:45] <_joe_>	 tltaylor: oh ok then I don't have a good answer sorry :)
[21:32:03] <_joe_>	 legoktm: uh possible yes
[21:32:22] <_joe_>	 they still need to serve the temporary domain cert
[21:32:37] <sukhe>	 I assumed that that was the case
[21:32:44] <sukhe>	 so I am not sure if I missed something but I have merged the DNS patch
[21:32:51] <sukhe>	 if there is more, happy to do that
[21:33:14] <Krinkle>	 tltaylor: normally the respective teams (design, sec, perf) would briefly consult in those aspects during planning and then confirm prelaunch review, as for anything else technical we do as an organisation.
[21:35:22] <legoktm>	 it seems to have a valid cert now
[21:35:40] <sukhe>	 yep
[21:35:51] <Krinkle>	 tltaylor: I didn't realise this was a private site. I don't know exactly what that means, but that probably obviates most vertical concerns. In the past it appeared as if planning processes intentionally avoiding involving Tech inc to eg no review of technical vendor or guiding planned work, and no preview stage shared, which inevitably the led to stress and post-hiv embarrassment, conflict and various teams helping out to fix up the 
[21:35:51] <Krinkle>	 work.
[21:36:44] <Krinkle>	 post-hoc*
[21:39:47] <_joe_>	 tltaylor: the site is now reachable :)
[21:40:13] <tltaylor>	 Comms will be appropriately thankful
[21:41:43] <tltaylor>	 I'll follow up on the other issues separately