[10:23:37] FYI, there are some dirty changes in helmfile.d/services/thumbor on deploy2002 (jayme?) [10:24:43] looks like I'm responsible for all dirty changes there now :-p [10:25:02] wasn't me in this case, though. hnowlan maybe? [10:25:24] oops, yep! I'll clean them up [10:25:48] ottomata: you should be good to go [13:55:47] 10serviceops, 10SRE: Migrate node-based services in production to node14 - https://phabricator.wikimedia.org/T306995 (10Lucas_Werkmeister_WMDE) [13:57:00] 10serviceops, 10SRE: Migrate node-based services in production to node14 - https://phabricator.wikimedia.org/T306995 (10Lucas_Werkmeister_WMDE) [13:57:08] 10serviceops: Migrate node-based services in production to node16 - https://phabricator.wikimedia.org/T308371 (10Lucas_Werkmeister_WMDE) [13:57:12] 10serviceops: Migrate node-based services in production to node16 - https://phabricator.wikimedia.org/T308371 (10Lucas_Werkmeister_WMDE) [14:07:05] 10serviceops: Migrate node-based services in production to node16 - https://phabricator.wikimedia.org/T308371 (10Clement_Goubert) FYI, we rebuilt the `nodejs16-slim` image to have a symlink from `/bin/node` to `/bin/nodejs` to have backwards compatibility with the former node images without having to change your... [14:17:33] 10serviceops, 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 10), 10Patch-For-Review, 10Service-deployment-requests: New Service Request mediawiki-page-content-change-enrichment - https://phabricator.wikimedia.org/T330507 (10JArguello-WMF) [14:22:43] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: March 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T327920 (10Clement_Goubert) [14:23:19] 10serviceops, 10Datacenter-Switchover, 10Patch-For-Review: switch noc.wikimedia.org from eqiad to codfw - https://phabricator.wikimedia.org/T331634 (10Clement_Goubert) [14:25:20] 10serviceops, 10Datacenter-Switchover, 10Patch-For-Review: switch noc.wikimedia.org from eqiad to codfw - https://phabricator.wikimedia.org/T331634 (10Clement_Goubert) 05In progress→03Resolved [14:25:30] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: March 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T327920 (10Clement_Goubert) [14:26:53] 10serviceops: Migrate node-based services in production to node16 - https://phabricator.wikimedia.org/T308371 (10Jdforrester-WMF) [14:31:02] 10serviceops, 10Datacenter-Switchover: switch noc.wikimedia.org from eqiad to codfw - https://phabricator.wikimedia.org/T331634 (10Clement_Goubert) Added documentation in https://wikitech.wikimedia.org/wiki/Switch_Datacenter#Manual_switch [15:15:35] 10serviceops: Migrate node-based services in production to node16 - https://phabricator.wikimedia.org/T308371 (10Jdforrester-WMF) [15:16:09] 10serviceops, 10Data-Engineering, 10Data-Persistence, 10Discovery-Search, and 7 others: eqiad row C switches upgrade - https://phabricator.wikimedia.org/T331882 (10ayounsi) [15:17:13] 10serviceops, 10DBA, 10Data-Engineering, 10Discovery-Search, and 7 others: eqiad row C switches upgrade - https://phabricator.wikimedia.org/T331882 (10Marostegui) [15:18:59] 10serviceops, 10Foundational Technology Requests, 10Prod-Kubernetes, 10Shared-Data-Infrastructure, and 2 others: Post Kubernetes v1.23 cleanup - https://phabricator.wikimedia.org/T328291 (10JMeybohm) [15:20:29] 10serviceops, 10DBA, 10Data-Engineering, 10Discovery-Search, and 7 others: eqiad row C switches upgrade - https://phabricator.wikimedia.org/T331882 (10Marostegui) [15:21:52] 10serviceops, 10DBA, 10Data-Engineering, 10Discovery-Search, and 7 others: eqiad row C switches upgrade - https://phabricator.wikimedia.org/T331882 (10Marostegui) Moved dbproxy1018 as it belongs to #cloud-services-team [15:22:12] 10serviceops, 10DBA, 10Data-Engineering, 10Discovery-Search, and 7 others: eqiad row C switches upgrade - https://phabricator.wikimedia.org/T331882 (10Marostegui) [15:22:30] 10serviceops, 10DBA, 10Data-Engineering, 10Discovery-Search, and 7 others: eqiad row C switches upgrade - https://phabricator.wikimedia.org/T331882 (10Marostegui) [15:22:55] 10serviceops, 10DBA, 10Data-Engineering, 10Discovery-Search, and 7 others: eqiad row C switches upgrade - https://phabricator.wikimedia.org/T331882 (10MoritzMuehlenhoff) [15:24:27] 10serviceops, 10DBA, 10Data-Engineering, 10Discovery-Search, and 7 others: eqiad row C switches upgrade - https://phabricator.wikimedia.org/T331882 (10Marostegui) [16:14:02] 10serviceops, 10Foundational Technology Requests, 10Prod-Kubernetes, 10Shared-Data-Infrastructure, and 2 others: Post Kubernetes v1.23 cleanup - https://phabricator.wikimedia.org/T328291 (10JMeybohm) [16:22:04] 10serviceops, 10SRE: ICU transition towards ICU 67 - https://phabricator.wikimedia.org/T329491 (10MoritzMuehlenhoff) These packages have been rebuilt against the ICU67-enabled PHP packages and imported to the component/icu67 component (some packages depend on others, e.g. igbinary on apcu and memcached on igbi... [16:26:10] 10serviceops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row C switches upgrade - https://phabricator.wikimedia.org/T331882 (10MPhamWMF) [16:27:17] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Make more use of Calico network policy features - https://phabricator.wikimedia.org/T331894 (10JMeybohm) [16:31:52] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Re-enable seccomProfile in cert-manager chart after k8s 1.23 migration completed - https://phabricator.wikimedia.org/T325620 (10JMeybohm) Related revert is at: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/896128 [16:50:13] 10serviceops, 10Prod-Kubernetes, 10Wikidata, 10Wikidata-Query-Service, 10wdwb-tech: Write and adapt Runbooks and cookbooks related to the WDQS Streaming Updater and kubernetes - https://phabricator.wikimedia.org/T293063 (10Gehel) [16:51:13] 10serviceops, 10Foundational Technology Requests, 10Prod-Kubernetes, 10Shared-Data-Infrastructure, and 2 others: Update Kubernetes clusters to v1.23 - https://phabricator.wikimedia.org/T307943 (10JMeybohm) [16:51:27] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Update cert-manager to 1.10.x - https://phabricator.wikimedia.org/T325292 (10JMeybohm) 05Resolved→03Open I managed to completely not upgrade cert-manager on all clusters as we have a global version override in `helmfile.d/admin_ng/cert-... [19:24:15] 10serviceops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row C switches upgrade - https://phabricator.wikimedia.org/T331882 (10colewhite) [20:57:53] 10serviceops, 10Arc-Lamp, 10Performance-Team (Radar), 10SRE Observability (FY2022/2023-Q3), 10User-fgiunchedi: Expand RAM on arclamp hosts and move them to baremetal - https://phabricator.wikimedia.org/T316223 (10Krinkle)