[08:14:25] 10serviceops, 10SRE, 10Traffic, 10Abstract Wikipedia team (Phase λ – Launch), 10HTTPS: Get new edge & internal HTTPS certificates expanded to add wikifunctions.org and *.wikifunctions.org - https://phabricator.wikimedia.org/T313227 (10Clement_Goubert) [08:15:27] 10serviceops, 10SRE, 10Traffic, 10Abstract Wikipedia team (Phase λ – Launch), 10HTTPS: Get new edge & internal HTTPS certificates expanded to add wikifunctions.org and *.wikifunctions.org - https://phabricator.wikimedia.org/T313227 (10Clement_Goubert) New internal certs now include `wikifunctions.org` an... [10:49:10] 10serviceops, 10MW-on-K8s, 10Recommendation-API, 10Patch-For-Review: Migrate recommendation-api to mw-api-int - https://phabricator.wikimedia.org/T334062 (10Clement_Goubert) ` cgoubert@deploy1002:/srv/deployment-charts/helmfile.d/services/recommendation-api$ service-checker-swagger staging.svc.eqiad.wmnet... [10:51:28] 10serviceops, 10MW-on-K8s, 10Recommendation-API, 10Patch-For-Review: Migrate recommendation-api to mw-api-int - https://phabricator.wikimedia.org/T334062 (10Clement_Goubert) ` cgoubert@deploy1002:/srv/deployment-charts/helmfile.d/services/recommendation-api$ service-checker-swagger recommendation-api.svc.c... [10:58:25] 10serviceops, 10MW-on-K8s, 10Recommendation-API, 10Patch-For-Review: Migrate recommendation-api to mw-api-int - https://phabricator.wikimedia.org/T334062 (10Clement_Goubert) ` cgoubert@deploy1002:/srv/deployment-charts/helmfile.d/services/recommendation-api$ service-checker-swagger recommendation-api.svc.e... [11:04:20] 10serviceops, 10MW-on-K8s, 10Recommendation-API, 10Patch-For-Review: Migrate recommendation-api to mw-api-int - https://phabricator.wikimedia.org/T334062 (10Clement_Goubert) 05Open→03Resolved Requests have now completely crossed over from `mwapi-async` (baremetal) to `mw-api-int-async` (k8s). Resolvin... [11:04:35] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, and 3 others: Migrate internal traffic to k8s - https://phabricator.wikimedia.org/T333120 (10Clement_Goubert) [11:04:46] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, and 3 others: Migrate internal traffic to k8s - https://phabricator.wikimedia.org/T333120 (10Clement_Goubert) [12:22:57] should the deploy.sh script in deployment-charts be removed? I was tempted to use it but saw that it uses the old helmfile layout (with an $env directory) [12:23:23] also, is there a shortcut yall have to deploy in all 3 wikikube envs? or do you do it manually? [12:23:30] asking for https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/914768 [12:23:42] which is 9 different helmfile apply comands [12:23:43] :) [12:24:33] oh actually 12i forgot main :) [12:30:38] There is no shortcut I think as there is an intentional fence between staging and production [12:33:22] the script, AIUI, was introduced by releng at some point (https://phabricator.wikimedia.org/T305033) [12:34:13] aye okay [12:35:39] huhm, any reason helmfile -e codfw apply would hang (eventgate-logging-external)? [12:35:42] events have things like [12:35:45] 58s Warning FailedScheduling pod/eventgate-canary-74f94d7fbb-45zqr 0/22 nodes are available: 13 node(s) were unschedulable, 2 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 3 Insufficient cpu, 4 node(s) had taint {dedicated: kask}, that the pod didn't tolerate. [12:39:15] yeah, I think reboots are running in codfw currently [12:39:58] that process will mark nodes as unshedulable, so you end up with less "capacity" [12:41:52] easiest to come back later for codfw, if that's an option [12:43:34] okay i can do that [12:43:47] i'll just wait for all and do later so I don't forget any [12:51:10] you may ask claime to ping you when reboots are done in codfw [13:03:19] Hi, yeah, I'm running reboots in codfw, I have 10 left [13:03:40] I'll ping when done [13:06:40] <3 [13:09:14] 10serviceops, 10SRE, 10Traffic, 10envoy: Upgrade Envoy to supported version - https://phabricator.wikimedia.org/T300324 (10JMeybohm) [13:09:25] 10serviceops, 10SRE, 10Traffic, 10envoy: Upgrade Envoy to supported version - https://phabricator.wikimedia.org/T300324 (10JMeybohm) [13:09:36] 10serviceops, 10SRE, 10Traffic, 10envoy: Upgrade Envoy to supported version - https://phabricator.wikimedia.org/T300324 (10JMeybohm) [13:13:13] 10serviceops, 10MW-on-K8s, 10Wikidata, 10Wikidata-Termbox, and 2 others: Migrate termbox to mw-api-int - https://phabricator.wikimedia.org/T334064 (10Clement_Goubert) There is a discussion to be had on whether we wish to merge these two environments (staging and staging-test) or keep the status quo. Holdin... [13:13:23] 10serviceops, 10MW-on-K8s, 10Wikidata, 10Wikidata-Termbox, and 2 others: Migrate termbox to mw-api-int - https://phabricator.wikimedia.org/T334064 (10Clement_Goubert) 05Open→03Stalled [13:13:30] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, and 3 others: Migrate internal traffic to k8s - https://phabricator.wikimedia.org/T333120 (10Clement_Goubert) [13:14:04] wow _joe_ deployment-charts CI is WAY faster. thank you so much! [13:20:11] 10serviceops, 10SRE, 10Traffic, 10envoy: Upgrade Envoy to supported version - https://phabricator.wikimedia.org/T300324 (10JMeybohm) [13:20:41] 10serviceops, 10Traffic, 10envoy: Upgrade Envoy to supported version - https://phabricator.wikimedia.org/T300324 (10JMeybohm) a:05RLazarus→03JMeybohm [13:45:13] 10serviceops, 10SRE, 10Traffic, 10envoy, 10Patch-For-Review: Refactor envoy HTTP protocol options to new version - https://phabricator.wikimedia.org/T303230 (10JMeybohm) a:05RLazarus→03JMeybohm [13:45:59] 10serviceops, 10SRE, 10envoy: Refactor envoy max_requests_per_connection from Cluster to HttpProtocolOptions - https://phabricator.wikimedia.org/T304124 (10JMeybohm) a:05RLazarus→03None [13:46:43] 10serviceops, 10Traffic, 10envoy, 10Patch-For-Review: Refactor envoy access_log_path to access loggers - https://phabricator.wikimedia.org/T303231 (10JMeybohm) a:05RLazarus→03JMeybohm [13:49:45] ottomata: Done with codfw [13:49:53] You can go ahead [14:03:41] ty [16:20:39] I'd like to e̶x̶p̶l̶o̶d̶e̶ ̶k̶8̶s̶ expand thumbor capacity as per our previous chats about capacity in k8s https://gerrit.wikimedia.org/r/914737