[04:41:14] 10serviceops, 10Data-Engineering, 10SRE, 10Traffic, 10Trust-and-Safety: Disable GeoIP Legacy Download - https://phabricator.wikimedia.org/T303464 (10odimitrijevic) [06:58:14] 10serviceops, 10ops-eqiad: mc1053 PS redundancy alert - https://phabricator.wikimedia.org/T304477 (10elukey) [07:47:51] hello folks [07:47:56] going to reimage kubernetes1009 [08:31:32] 10serviceops: PHP Notice: Constant PHP_VERSION already defined in /var/www/php-monitoring/lib.php on line 4 - https://phabricator.wikimedia.org/T301945 (10hashar) [08:40:42] 10serviceops, 10Patch-For-Review: PHP Notice: Constant PHP_VERSION already defined in /var/www/php-monitoring/lib.php on line 4 - https://phabricator.wikimedia.org/T301945 (10hashar) a:03hashar [08:43:47] kubernetes1009 done [10:17:02] 10serviceops, 10GitLab: Standardize Debian package builds on GitLab CI - https://phabricator.wikimedia.org/T304491 (10Jelto) [10:18:35] 10serviceops, 10Security-Team, 10GitLab (CI & Job Runners), 10Patch-For-Review, and 2 others: Setup GitLab Runner in trusted environment - https://phabricator.wikimedia.org/T295481 (10Jelto) [10:21:51] 10serviceops: PHP Notice: Constant PHP_VERSION already defined in /var/www/php-monitoring/lib.php on line 4 - https://phabricator.wikimedia.org/T301945 (10hashar) 05Open→03Resolved I have verified the log spam is entirely gone from logstash. Thanks! [10:25:55] 10serviceops, 10Security-Team, 10GitLab (CI & Job Runners), 10Patch-For-Review, and 2 others: Setup GitLab Runner in trusted environment - https://phabricator.wikimedia.org/T295481 (10Jelto) I mirrored [wmf-sre-laptop](https://gitlab.wikimedia.org/repos/sre/wmf-sre-laptop) to GitLab and created a very basi... [11:01:05] 10serviceops, 10MediaWiki-extensions-PropertySuggester, 10Wikidata, 10wdwb-tech, 10Service-deployment-requests: New Service Request SchemaTree - https://phabricator.wikimedia.org/T301471 (10Michaelcochez) @QChris I noticed the addition of the .gitreview file on gerrit. Is this file needed? If so, we woul... [11:25:48] 10serviceops, 10GitLab: Standardize Debian package builds on GitLab CI - https://phabricator.wikimedia.org/T304491 (10Jelto) [11:46:12] 10serviceops, 10SRE, 10Thumbor, 10Service-deployment-requests: New Service Request Wikimedia-Thumbor - https://phabricator.wikimedia.org/T304436 (10jbond) p:05Triage→03Medium [11:46:41] 10serviceops, 10SRE, 10ops-eqiad: mc1053 PS redundancy alert - https://phabricator.wikimedia.org/T304477 (10jbond) p:05Triage→03Medium [13:12:52] hello again folks, reimaging kubernetes1010 [13:55:21] done, 5 nodes left [15:16:21] 10serviceops, 10Data-Catalog, 10Data-Engineering, 10SRE, and 2 others: New Service Request: DataHub - https://phabricator.wikimedia.org/T303049 (10JMeybohm) For the Ingress part we will need to use two different names/discovery records for the services (as we can't distinguish by port). Maybe `datahub.disc... [15:16:35] 10serviceops, 10Data-Catalog, 10Data-Engineering, 10SRE, and 2 others: New Service Request: DataHub - https://phabricator.wikimedia.org/T303049 (10JMeybohm) a:03JMeybohm [15:21:30] 10serviceops, 10Data-Catalog, 10Data-Engineering, 10SRE, and 2 others: New Service Request: DataHub - https://phabricator.wikimedia.org/T303049 (10BTullis) >>! In T303049#7800172, @JMeybohm wrote: > For the Ingress part we will need to use two different names/discovery records for the services (as we can't... [15:27:59] 10serviceops, 10Data-Catalog, 10Data-Engineering, 10SRE, and 2 others: New Service Request: DataHub - https://phabricator.wikimedia.org/T303049 (10JMeybohm) >>! In T303049#7800187, @BTullis wrote: > It's not going to affect the public-facing (but authenticated) URL of https://datahub.wikimedia.org for the... [15:32:43] 10serviceops, 10Wikimedia-Developer-Portal, 10Goal, 10Service-deployment-requests: New Service Request: developer-portal - https://phabricator.wikimedia.org/T297140 (10Majavah) What hostname will this be hosted on? [15:38:16] 10serviceops, 10Wikimedia-Developer-Portal, 10Goal, 10Service-deployment-requests: New Service Request: developer-portal - https://phabricator.wikimedia.org/T297140 (10bd808) >>! In T297140#7800215, @Majavah wrote: > What hostname will this be hosted on? developer.wikimedia.org is the preferred hostname (... [16:05:07] I have some time so I'd go for kubernetes1011 as well if everybody is ok [16:10:24] started [16:30:53] 10serviceops, 10MediaWiki-extensions-PropertySuggester, 10Wikidata, 10wdwb-tech, 10Service-deployment-requests: New Service Request SchemaTree - https://phabricator.wikimedia.org/T301471 (10Michaelcochez) @Joe : would it be fine if we include the index file into the image for the A/B testing phase? If th... [17:14:57] 10serviceops, 10SRE: Service puppet certificate due to expire - https://phabricator.wikimedia.org/T304543 (10jbond) p:05Triage→03High [17:15:47] hi all i just created https://phabricator.wikimedia.org/T304543 the svc certificates for api, appservers and render are al due to expire on the monday (28/03/2022) [17:21:08] jbond: see also https://phabricator.wikimedia.org/T304237 [17:21:35] they should already be renewed, did something get missed? [17:22:53] * jbond looking [17:23:11] 1012 reimaged and serving traffic [17:23:15] err 1011 [17:24:17] rzl: ack looks like we just need to do this bit https://phabricator.wikimedia.org/T304237#7797398 will follow up with jayme thanks [17:24:33] jbond: cool, thank you for noticing it! [17:25:05] 10serviceops, 10SRE: Service puppet certificate due to expire - https://phabricator.wikimedia.org/T304543 (10jbond) [17:25:35] 10serviceops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Cert renewal for {appserver,api}.svc.{eqiad,codfw}.wmnet - https://phabricator.wikimedia.org/T304237 (10jbond) [17:26:02] 10serviceops, 10SRE, 10Traffic, 10envoy, 10Patch-For-Review: Upgrade Envoy to supported version - https://phabricator.wikimedia.org/T300324 (10RLazarus) [17:26:10] 10serviceops, 10Beta-Cluster-Infrastructure, 10SRE, 10envoy, 10Patch-For-Review: Clean up Puppet support for Envoy v2 config API - https://phabricator.wikimedia.org/T303770 (10RLazarus) 05Open→03Resolved [17:28:50] 10serviceops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Cert renewal for {appserver,api}.svc.{eqiad,codfw}.wmnet - https://phabricator.wikimedia.org/T304237 (10jbond) >>! In T304237#7797420, @Volans wrote: >>>! In T304237#7797398, @JMeybohm wrote: >>>>! In T304237#7795994, @Volans wrote: >>>... [17:31:23] rzl: np :) [18:49:04] 10serviceops, 10SRE, 10Traffic, 10envoy, 10Patch-For-Review: Upgrade Envoy to supported version - https://phabricator.wikimedia.org/T300324 (10RLazarus) Hmm, the 1.21.1 build didn't work out of the box. Running `build-envoy-deb buster future` got me this: ` [...] ./ci/run_envoy_docker.sh ./ci/do_ci.sh b... [23:40:36] 10serviceops, 10SRE, 10Traffic, 10envoy, 10Patch-For-Review: Upgrade Envoy to supported version - https://phabricator.wikimedia.org/T300324 (10RLazarus) [23:40:46] 10serviceops, 10SRE, 10Traffic, 10envoy, 10Patch-For-Review: Refactor envoy HTTP protocol options to new version - https://phabricator.wikimedia.org/T303230 (10RLazarus) 05Stalled→03In progress p:05Low→03Medium