[00:01:41] Ctrl-a followed by num [08:55:25] headsup: I'm going to deploy a puppet change that generates netboot.cfg using a template instead of having it be hand crafted (https://gerrit.wikimedia.org/r/c/operations/puppet/+/973308). We checked the diff between the original and generated file, and didn't see any impacful one. If you see anything weird related to re-image and partman recipes, [08:55:25] scream, It's probably my fault. [09:31:20] nicely done brouberol ! [09:34:27] silly question; if I'm making a puppet change that affects one host in particular (due to a hiera change to that host), should I expect Hosts: auto to include that host? [09:37:59] <_joe_> Emperor: I don't think so if it's a hiera change [09:38:07] <_joe_> unless john and jesse added that [09:38:31] FE :) [09:38:34] <_joe_> brouberol: please update the documentation :) [09:39:20] <_joe_> Emperor: basically it's overly complicated to do hiera -> host affected without actually doing a hiera lookup, so it's not a simple problem to solve [09:39:33] <_joe_> hiera file => list of affected hosts [09:39:45] <_joe_> it's doable, it just requires time that I never had :) [09:39:56] Mmm (and it was easy enough to write a Hosts: line, I was just a bit surprised :) [09:57:41] _joe_: done, I updated https://wikitech.wikimedia.org/wiki/Server_Lifecycle [10:04:24] <_joe_> brouberol: great [10:15:10] I'm merging thanos.w.o OIDC/SSO support, you'll be logged out and shown the oauth2-proxy splash screen [10:15:25] logged out of thanos.w.o that is, not sso [10:22:05] aannd it didn't work as expected, investigating [10:24:09] and we're back [10:24:59] godog: the page just arrived I imagine is about this ^^, correct? [10:25:29] fabfur: yes, double checking [10:25:32] sorry about that [10:25:56] np :) [10:26:04] I'll ack it [10:26:12] thank you, but yes should be recovering soon [10:26:17] I accidentally apache [10:30:26] <_joe_> I didn't know you could use that as a verb [10:32:05] this is what I had in mind https://knowyourmeme.com/memes/i-accidentally [10:32:33] _joe_: verbing weirds language [10:32:55] <_joe_> godog: yeah but it was more interesting if "to apache" meant "to mess up" [10:33:01] <_joe_> which seemed appropriate [10:33:49] harsh [10:36:52] <_joe_> Emperor: as someone who had to read the httpd source code, and in particular that impressive spaghetti bowl that is mod_proxy_fcgi.c, I think the term you were searching for is "fair" [10:36:56] <_joe_> :P [10:37:41] <_joe_> (also, somehow even after 15 years and extensive study, I still am never sure the apache config change I made does what intended with 100% confidence, which is amazing if you think about it) [11:05:15] harsh but fair, then? :) [11:06:01] (definitely not my default mode of interaction) [13:49:02] https://github.com/p8952/bocker WCPGW? docker implemented as a shell script [14:23:39] "Bocker" sounds like a Muppet I don't want to know ;) [14:28:28] cwhite, fabfur, jbond, on-call head's up, I'm going to raise mw-on-k8s from 15% to 20% of global traffic. Scaling up of mw-web and mw-api-ext has been done, so there should not be any issue. Ping me or joe if there's a problem. [14:28:43] ack thanks claime [14:29:02] hi claime we're resolving an incident in esams, are you sure? [14:29:13] How do emails to alerts@ mentioning p.ages relate to VO incidents? [e.g. I got a few about cr1-esams just now] [14:29:14] jbond: godog: moritzm: not sure if I should reply on https://phabricator.wikimedia.org/T351181, because that task has been closed, but you should consider implementing https://phabricator.wikimedia.org/T324623 (I used the openssl driver for Cloud VPS) [14:29:19] ah ok, sorry if jbond agrees no prob on my side! [14:29:21] fabfur: No problem re: esams [14:29:48] well maybe leave it 30 mins [14:30:17] however we know the cause of this incident and it unlikley to happen again so we are pretty safe [14:30:33] (fabfur claime ^^) [14:31:01] yes, and the incident seems closed [14:31:13] jbond: err clicked submit before I saw your message jbond, it's a trafficserver lua config change, I don't think it'll be affected by a link being down [14:32:24] claime: i think its fine [14:32:31] thanks [14:32:52] ack, thx [14:33:31] andrewbogott: Can I merge your cloud partman changes? [14:33:34] yes please [14:33:35] claime: yeah we are stable now, have some ports down but it's fine, just loss of redundancy, your change is ok [14:33:42] topranks: fantastic, thank you [14:33:46] claime: I wasn't feeling great about merging yours :) [14:34:01] andrewbogott: Understandable :D [14:34:34] Merged [14:56:14] Southparkfan: totally, thank you for that work! we definitely want to move to openssl as moritzm was mentioning [14:57:05] Southparkfan: please comment on task, tHat's useful context :-) [14:57:49] and I had no idea about https://phabricator.wikimedia.org/T324623. then I don't even need to create a task [14:58:31] you've built me a package back in the days moritz, maybe you've already forgotten ;) [14:58:53] I'll reply on the task, of course help from others is needed to perform the actual migration.. [15:09:31] We're at 20% of external traffic to mw-on-k8s, and everything looks good. [15:09:47] I had definititely already forgotten, but now that you mention it, some memory comes back :-) [22:40:17] getting a puppet CRL expiration error on cloduelastic1007, has anyone seen this before? [22:48:07] probably related to the puppet 7 migration? [22:48:37] this is a host that's been up for about a month [22:57:25] inflatador: yes, ideally leave the screen/tmux running and tell John about it [22:58:26] inflatador: well, is it the "self-signed cert in chain", if not then it's different from my case [23:00:33] mutante I'm getting `Error: The CRL issued by 'CN=Wikimedia_Internal_Root_CA,OU=Cloud Services,O=Wikimedia Foundation\, Inc,L=San Francisco,ST=California,C=US' has expired, verify time is synchronized` on one host [23:00:52] And `Error: Could not download CA certificate: Bad Request` on another [23:01:36] I'm at the end of my day, so ryankemper and I are making a phab task for IF [23:28:31] slightly different but also similar in that the Cloud Services OU is in there [23:28:37] sounds good