[01:40:15] hi! Was https://phabricator.wikimedia.org/T320848 discussed yesterday? [02:54:59] 10serviceops, 10MW-1.40-notes (1.40.0-wmf.6; 2022-10-17), 10PHP 7.4 support, 10Patch-For-Review, 10Platform Team Workboards (Clinic Duty Team): Rename articles and users to update our case mapping to PHP 7.4 and Unicode 11 - https://phabricator.wikimedia.org/T292552 (10tstarling) The script is still runn... [04:22:01] 10serviceops, 10MW-1.40-notes (1.40.0-wmf.6; 2022-10-17), 10PHP 7.4 support, 10Patch-For-Review, 10Platform Team Workboards (Clinic Duty Team): Rename articles and users to update our case mapping to PHP 7.4 and Unicode 11 - https://phabricator.wikimedia.org/T292552 (10tstarling) I deduplicated the user... [04:46:23] 10serviceops, 10MW-1.40-notes (1.40.0-wmf.6; 2022-10-17), 10PHP 7.4 support, 10Patch-For-Review, 10Platform Team Workboards (Clinic Duty Team): Rename articles and users to update our case mapping to PHP 7.4 and Unicode 11 - https://phabricator.wikimedia.org/T292552 (10tstarling) I reinserted the 14 ⓝⓘⓙⓦ... [04:51:31] 10serviceops, 10MW-1.40-notes (1.40.0-wmf.6; 2022-10-17), 10PHP 7.4 support, 10Patch-For-Review, 10Platform Team Workboards (Clinic Duty Team): Rename articles and users to update our case mapping to PHP 7.4 and Unicode 11 - https://phabricator.wikimedia.org/T292552 (10tstarling) I used attachAccount.php... [05:27:53] 10serviceops, 10MW-1.40-notes (1.40.0-wmf.6; 2022-10-17), 10PHP 7.4 support, 10Patch-For-Review, 10Platform Team Workboards (Clinic Duty Team): Rename articles and users to update our case mapping to PHP 7.4 and Unicode 11 - https://phabricator.wikimedia.org/T292552 (10tstarling) Trying to clean up the r... [08:19:03] <_joe_> hnowlan: I remember back in the day we thought there wasn't much value in using secrets because any deployer could see all of them [08:19:12] <_joe_> but I guess with helm3 maybe that's changed? [08:19:17] <_joe_> actually, not really [08:25:32] _joe_: I was about to ask about some docker-pkg pending change only to find out you got it merged half an hour ago :-] [08:26:34] there is another straight forward one to have test.sh find /usr/local/bin/docker by amending PATH (that is for Mac OS) https://gerrit.wikimedia.org/r/c/operations/docker-images/docker-pkg/+/692995 [08:27:06] no, it did not change at all. But "if" we decide to use something proper to store secrets in the future, we are prepared if we already use the API for that [08:28:03] <_joe_> jayme: ok [08:39:21] hello folks, I got added to https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/845680 [08:39:49] I am not sure what is the best direction of it, can somebody follow up? [08:40:36] <_joe_> yeah, it's a good patch which might soon become outdated [08:40:48] <_joe_> although I resisted adding a gemfile for $reasons [08:40:56] <_joe_> I can comment on the patch [08:42:55] thanks :) [08:51:08] Can I get a review for https://gerrit.wikimedia.org/r/c/operations/puppet/+/844991 pretty please ? [08:52:45] <_joe_> btullis: ^^ sometimes you also have to ping me multiple times, see? [08:53:43] <_joe_> again all of you: don't worry about pressuring me, if I don't have time I'll tell you [08:54:21] _joe_: Yes, understood. I'll be more vocal in future. [09:29:09] 10serviceops, 10SRE, 10Traffic, 10Abstract Wikipedia team (Phase λ – Launch), and 2 others: Get new edge & internal HTTPS certificates expanded to add wikifunctions.org and *.wikifunctions.org - https://phabricator.wikimedia.org/T313227 (10Vgutierrez) acme-chief will deploy the unified cert shipping `wikif... [09:42:22] 10serviceops, 10SRE, 10Performance-Team (Radar): Phase out nutcracker from mediawiki servers - https://phabricator.wikimedia.org/T277183 (10Clement_Goubert) [09:42:30] 10serviceops, 10SRE, 10Performance-Team (Radar): Remove nutcracker from mediawiki chart - https://phabricator.wikimedia.org/T321042 (10Clement_Goubert) 05Open→03Resolved [10:01:20] 10serviceops, 10SRE, 10Traffic, 10Abstract Wikipedia team (Phase λ – Launch), and 2 others: Get new edge & internal HTTPS certificates expanded to add wikifunctions.org and *.wikifunctions.org - https://phabricator.wikimedia.org/T313227 (10Vgutierrez) [11:18:41] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Deploy new mw-debug service - https://phabricator.wikimedia.org/T321201 (10Clement_Goubert) p:05Triage→03Medium [13:43:30] good morning [13:43:41] _joe_: was https://phabricator.wikimedia.org/T320848 discussed at the serviceops meeting? [13:43:56] <_joe_> legoktm: yes, we should talk [13:44:13] <_joe_> but not right now [13:44:35] <_joe_> I'm finishing up fixing logging in shellbox [13:44:51] ok :) [13:58:37] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Import coredns 1.8.x (k8s 1.23 dependency) - https://phabricator.wikimedia.org/T321159 (10elukey) [14:44:50] teeny review for adding thumbor to k8s deployment services https://gerrit.wikimedia.org/r/c/operations/puppet/+/849591 [15:40:44] from a k8s perspective for certificate generation etc (which I assume won't be an issue as thumbor is currently unencrypted - but that will actually pose issues for us if we're planning on doing a mixed migration between old thumbor nodes and k8s nodes...) what name should I be using? Should there be a thumbor-k8s or should I be creating certificates for an eventual k8s-based [15:40:50] thumbor.discovery.wmnet? [15:41:55] are we going to funnel it through ingress? [15:41:57] <_joe_> your cert needs to be reachable with the name swift uses [15:41:58] jayme ^ [15:42:08] <_joe_> akosiaris: I don't think that's advisable tbh [15:42:23] capacity wise? [15:42:46] <_joe_> akosiaris: I personally don't trust the istio ingress enough to hand it the keys to our thumbnailing [15:43:00] <_joe_> both us and el.ukey saw some bugs already [15:44:08] yeah, we agreed on not doing it for now because of our lack of experience [15:44:39] until now I unfortunately did not manage to pull some more "small" services behind ingress :/ [15:46:11] gtg for today, ttyl o/ [15:46:33] <_joe_> it would be great if we could use cert-manager for every service though [15:46:35] so, thumbor doesn't have a discovery record [15:46:51] probably cause of being such a straggler ? [15:47:03] <_joe_> hnowlan: wait a sec, I would assume that for the transition you'd need to proceed without encryption [15:47:14] <_joe_> akosiaris: thumbor is an appendage of swift [15:47:22] <_joe_> the discovery records make sense for swift [15:47:29] <_joe_> and thumbor follows [15:47:42] <_joe_> no traffic to swift => no traffic to thumbor more or less [15:47:47] that sounds like hard coupling though [15:48:00] we might want to depool thumbor in a site [15:48:03] <_joe_> well it is, but sure we can move away from that model eventually [15:48:04] for whatever reasons [15:48:09] <_joe_> just not right now [15:48:11] <_joe_> :) [15:48:12] _joe_: yep, seems like the only option [15:48:30] <_joe_> right now I would first make the transition, then we can plan what we want to improve there [15:49:52] yeah, the usual need to balance between not scope creeping the project and knowing that we won't have another chance to work on it for years [15:56:58] related :) https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/849605 [15:57:55] +1'd, I was lurking [15:59:19] Well then you go ahead and reset that why don't you hnowlan :p [16:00:03] hehh soz [16:00:15] ;) [16:28:03] 10serviceops, 10Shellbox, 10SyntaxHighlight: Install pygments in Shellbox container with pip, not a Debian package - https://phabricator.wikimedia.org/T320848 (10bd808) > From a security perspective, we should build it from the sdist (pinned to the hash) and not use upstream wheels. As far as I know we do n... [16:47:25] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install contint1002 - https://phabricator.wikimedia.org/T313830 (10wiki_willy) a:03Cmjohnson [17:03:19] 10serviceops, 10Foundational Technology Requests, 10Prod-Kubernetes, 10Shared-Data-Infrastructure, and 2 others: Update Kubernetes clusters to v1.23 - https://phabricator.wikimedia.org/T307943 (10DAbad) [19:43:08] 10serviceops, 10API Platform (Sprint 00), 10Platform Team Workboards (Platform Engineering Reliability): New Service Request uniqueDevices Endpoint: AQS 2.0 - https://phabricator.wikimedia.org/T320967 (10VirginiaPoundstone) [21:09:16] I see that some k8s tokens are duplicated between disparate clusters, e.g. profile::calico::kubernetes::calicoctl::token, is that intentional? [22:32:54] 10serviceops, 10MW-1.40-notes (1.40.0-wmf.6; 2022-10-17), 10PHP 7.4 support, 10Patch-For-Review, 10Platform Team Workboards (Clinic Duty Team): Rename articles and users to update our case mapping to PHP 7.4 and Unicode 11 - https://phabricator.wikimedia.org/T292552 (10tstarling) * LJ: done with Special... [22:46:46] 10serviceops, 10MW-1.40-notes (1.40.0-wmf.6; 2022-10-17), 10PHP 7.4 support, 10Patch-For-Review, 10Platform Team Workboards (Clinic Duty Team): Rename articles and users to update our case mapping to PHP 7.4 and Unicode 11 - https://phabricator.wikimedia.org/T292552 (10tstarling) [22:47:51] 10serviceops, 10MW-1.40-notes (1.40.0-wmf.6; 2022-10-17), 10PHP 7.4 support, 10Patch-For-Review, 10Platform Team Workboards (Clinic Duty Team): Rename articles and users to update our case mapping to PHP 7.4 and Unicode 11 - https://phabricator.wikimedia.org/T292552 (10tstarling) Note that fixStuckGlobal... [23:08:53] 10serviceops, 10MW-1.40-notes (1.40.0-wmf.6; 2022-10-17), 10PHP 7.4 support, 10Patch-For-Review, 10Platform Team Workboards (Clinic Duty Team): Rename articles and users to update our case mapping to PHP 7.4 and Unicode 11 - https://phabricator.wikimedia.org/T292552 (10tstarling) * ῦΰῡΏ, ⅵ5lasevilsion, ⅹ... [23:46:33] 10serviceops, 10MW-1.40-notes (1.40.0-wmf.6; 2022-10-17), 10PHP 7.4 support, 10Patch-For-Review, 10Platform Team Workboards (Clinic Duty Team): Rename articles and users to update our case mapping to PHP 7.4 and Unicode 11 - https://phabricator.wikimedia.org/T292552 (10tstarling) * st, ſt: manually renamed