[00:03:19] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad, 10Patch-For-Review: Q3:(Need By: TBD) rack/setup/install conf100[789] - https://phabricator.wikimedia.org/T301272 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host conf1008.eqiad.wmnet with OS bullseye [00:03:37] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad, 10Patch-For-Review: Q3:(Need By: TBD) rack/setup/install conf100[789] - https://phabricator.wikimedia.org/T301272 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host conf1009.eqiad.wmnet with OS bullseye [00:25:03] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad, 10Patch-For-Review: Q3:(Need By: TBD) rack/setup/install conf100[789] - https://phabricator.wikimedia.org/T301272 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host conf1007.eqiad.wmnet with OS bullseye completed... [00:31:35] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q3:(Need By: TBD) rack/setup/install conf100[789] - https://phabricator.wikimedia.org/T301272 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host conf1009.eqiad.wmnet with OS bullseye completed: - conf1009 (**PASS**... [00:34:22] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q3:(Need By: TBD) rack/setup/install conf100[789] - https://phabricator.wikimedia.org/T301272 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host conf1008.eqiad.wmnet with OS bullseye completed: - conf1008 (**PASS**... [00:35:11] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q3:(Need By: TBD) rack/setup/install conf100[789] - https://phabricator.wikimedia.org/T301272 (10Cmjohnson) [00:35:18] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q3:(Need By: TBD) rack/setup/install conf100[789] - https://phabricator.wikimedia.org/T301272 (10Cmjohnson) 05Open→03Resolved [07:29:36] 10serviceops: Allow sending traffic to php 7.2 or 7.4 selectively in the apache configuration for MediaWiki - https://phabricator.wikimedia.org/T293451 (10Joe) a:03Joe [07:33:52] 10serviceops: Allow sending traffic to php 7.2 or 7.4 selectively in the apache configuration for MediaWiki - https://phabricator.wikimedia.org/T293451 (10Joe) 05Open→03Resolved [07:33:54] 10serviceops, 10Performance-Team (Radar): Migrate WMF production from PHP 7.2 to PHP 7.4 - https://phabricator.wikimedia.org/T271736 (10Joe) [07:34:36] 10serviceops: Allow sending traffic to php 7.2 or 7.4 selectively in the apache configuration for MediaWiki - https://phabricator.wikimedia.org/T293451 (10Joe) p:05Triage→03High [08:34:24] 10serviceops, 10Infrastructure-Foundations, 10netbox: Netbox and Redis - https://phabricator.wikimedia.org/T311385 (10ayounsi) p:05Triage→03Low [08:46:21] 10serviceops, 10Performance-Team (Radar): Install php 7.4 in production - https://phabricator.wikimedia.org/T311386 (10Joe) [08:53:08] 10serviceops, 10Performance-Team (Radar): Allow assigning each user to a specific php engine by setting a PHP_ENGINE cookie - https://phabricator.wikimedia.org/T311388 (10Joe) [08:53:47] 10serviceops, 10Performance-Team (Radar): Migrate WMF production from PHP 7.2 to PHP 7.4 - https://phabricator.wikimedia.org/T271736 (10Joe) p:05Triage→03High [08:55:42] 10serviceops, 10PHP 7.3 support, 10PHP 7.4 support, 10Platform Team Workboards (Clinic Duty Team): Rename articles and users to prepare for PHP 7.3 unicode changes - https://phabricator.wikimedia.org/T292552 (10Joe) p:05Triage→03High [08:59:15] 10serviceops, 10PHP 7.3 support, 10PHP 7.4 support, 10Platform Team Workboards (Clinic Duty Team): Rename articles and users to prepare for PHP 7.3 unicode changes - https://phabricator.wikimedia.org/T292552 (10Joe) I think it's reasonable to run the rename only once we've fully migrated to php 7.4. We wil... [09:09:07] 10serviceops, 10Performance-Team (Radar): Allow assigning each user to a specific php engine by setting a PHP_ENGINE cookie - https://phabricator.wikimedia.org/T311388 (10Joe) Important detail: we can either split the caches at the varnish layer as we did in the past, or just don't bother and just live with th... [09:09:16] 10serviceops, 10Performance-Team (Radar): Allow assigning each user to a specific php engine by setting a PHP_ENGINE cookie - https://phabricator.wikimedia.org/T311388 (10Joe) p:05Triage→03High [09:31:02] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Update Kubernetes clusters to v1.23 - https://phabricator.wikimedia.org/T307943 (10JMeybohm) [09:31:32] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Update Kubernetes clusters to v1.23 - https://phabricator.wikimedia.org/T307943 (10JMeybohm) [12:00:01] 10serviceops, 10Infrastructure-Foundations, 10netbox: Netbox and Redis - https://phabricator.wikimedia.org/T311385 (10akosiaris) Hi! Thanks for this task. So, from what I gathered, netbox uses Redis for caching and task queuing purposes, and support using different databases per function (queuing vs cachin... [12:44:42] 10serviceops: Put conf100[789] in production - https://phabricator.wikimedia.org/T311407 (10akosiaris) [12:47:12] 10serviceops: Decomission conf100[456] - https://phabricator.wikimedia.org/T311408 (10akosiaris) [12:47:30] 10serviceops: Decomission conf100[456] - https://phabricator.wikimedia.org/T311408 (10akosiaris) [12:47:32] 10serviceops: Put conf100[789] in production - https://phabricator.wikimedia.org/T311407 (10akosiaris) [12:47:56] 10serviceops: Put conf100[789] in production - https://phabricator.wikimedia.org/T311407 (10akosiaris) [12:48:01] 10serviceops, 10SRE, 10Patch-For-Review: Update conf1* servers - https://phabricator.wikimedia.org/T310062 (10akosiaris) [12:48:03] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q3:(Need By: TBD) rack/setup/install conf100[789] - https://phabricator.wikimedia.org/T301272 (10akosiaris) [14:53:36] 10serviceops, 10Patch-For-Review, 10Performance-Team (Radar): Install php 7.4 in production - https://phabricator.wikimedia.org/T311386 (10MoritzMuehlenhoff) I have updated our PHP build in component/php74 to the latest 7.4.x release; 7.4.30. Uploaded as 7.4.30-3+0~20220627.69+debian10~1.gbpf2b381+wmf1+buster1 [14:54:21] <_joe_> moritzm: <3 [15:20:17] 10serviceops, 10Wikimedia-production-error: fawiki user reports getting 503 errors with message "upstream connect error or disconnect before headers" - https://phabricator.wikimedia.org/T310450 (10Gehel) Removing the Search Platform team from this, it seems entirely unrelated to CirrusSearch. [15:50:13] 10serviceops, 10Data-Persistence-Backup, 10GitLab (Infrastructure), 10Patch-For-Review, 10User-brennen: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10LSobanski) @jelto Is there anything else that needs to happen here? [16:37:12] 10serviceops, 10Wikimedia-production-error: fawiki user reports getting 503 errors with message "upstream connect error or disconnect before headers" - https://phabricator.wikimedia.org/T310450 (10akosiaris) There is T301505 which points out `this is a symptom of a number of different problems that can appear... [17:35:55] 10serviceops, 10SRE: Provide node14 and node16 images for running production node-based services - https://phabricator.wikimedia.org/T306996 (10bd808) >>! In T306996#7914578, @bd808 wrote: >>>! In T306996#7912881, @Joe wrote: >> I've build and published the `nodejs14-slim` and the `nodejs16-slim` images, using... [17:59:22] 10serviceops, 10DNS, 10SRE, 10Traffic, and 3 others: Setup redirect of policy.wikimedia.org to Advocacy portal on Foundation website - https://phabricator.wikimedia.org/T310738 (10LSobanski) @Varnent could we get a clarification of the timeline for this request? The description says end of this month and y... [18:20:14] 10serviceops, 10GitLab (Infrastructure), 10Patch-For-Review: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by aokoth@cumin1001 for hosts: `gitlab2001.wikimedia.org` - gitlab2001.wikimedia.org (**F... [19:05:02] 10serviceops, 10Add-Link, 10Growth-Team: Investigate increase and fluctuation in max CPU for linkrecommendation-internal container - https://phabricator.wikimedia.org/T303177 (10MShilova_WMF) [20:43:48] 10serviceops, 10Infrastructure-Foundations, 10SRE, 10SRE-Access-Requests, and 3 others: Access to trusted gitlab runners for gitlab-roots (or appropriate similar group) - https://phabricator.wikimedia.org/T308350 (10brennen) 05Resolved→03Open While `contint-roots` members do now have access to `gitlab1... [21:51:23] 10serviceops, 10Infrastructure-Foundations, 10SRE, 10SRE-Access-Requests, and 5 others: Access to trusted gitlab runners for gitlab-roots (or appropriate similar group) - https://phabricator.wikimedia.org/T308350 (10Dzahn) [21:56:11] 10serviceops, 10Data-Persistence-Backup, 10serviceops-collab, 10GitLab (Infrastructure), and 2 others: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10Dzahn) [21:56:49] 10serviceops, 10Phabricator, 10serviceops-collab, 10Release-Engineering-Team (Next): Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10Dzahn) [21:57:01] 10serviceops, 10Phabricator, 10serviceops-collab, 10Patch-For-Review: move phabricator to new hardware generation - https://phabricator.wikimedia.org/T280597 (10Dzahn) [21:57:26] 10serviceops, 10Gerrit, 10SRE, 10serviceops-collab: replacement for gerrit2001 - https://phabricator.wikimedia.org/T243027 (10Dzahn)