[02:14:47] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install kubernetes10[27-56] - https://phabricator.wikimedia.org/T342533 (10Jhancock.wm) [07:10:01] 10serviceops, 10SRE, 10Patch-For-Review: Migrate conf2* hosts to bullseye - https://phabricator.wikimedia.org/T332010 (10JMeybohm) 05Resolved→03Open SRE was paged due to EtcdReplicationDown. Turns out the etcdmirror webinterface does not work with python3 on bullseye [07:57:31] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Move 10% of mediawiki external requests to mw on k8s - https://phabricator.wikimedia.org/T346422 (10Joe) [08:25:43] hello folks [08:26:00] I'd need to raise the changeprop's concurrency for ORESFetchScoreJob - https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/957864 [08:26:03] lemme know if it is ok or not [08:30:56] thank youuu [08:31:07] deploying cp-jobqueues [08:31:18] yw [08:48:41] deployed :) [09:27:28] 10serviceops, 10Data-Platform-SRE, 10Discovery-Search (Current work): Requesting permission to use kafka-main cluster to transport CirrusSearch updates - https://phabricator.wikimedia.org/T341625 (10Gehel) 05Open→03Resolved a:03Gehel Decision has been documented in https://wikitech.wikimedia.org/wiki/S... [09:27:32] 10serviceops: Rebalance kafka partitions in main-{eqiad,codfw} clusters - 2023 edition - https://phabricator.wikimedia.org/T341558 (10Gehel) [09:38:28] 10serviceops, 10SRE, 10CommRel-Specialists-Support (Jul-Sep-2023), 10Datacenter-Switchover: CommRel support for September 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T345265 (10Trizek-WMF) [09:40:38] 10serviceops, 10SRE, 10CommRel-Specialists-Support (Jul-Sep-2023), 10Datacenter-Switchover: CommRel support for September 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T345265 (10Trizek-WMF) [10:23:16] 10serviceops, 10collaboration-services, 10GitLab (CI & Job Runners): Standardize Debian package builds on GitLab CI - https://phabricator.wikimedia.org/T304491 (10MatthewVernon) Noting here that we've moved the wmf-debci repo to https://gitlab.wikimedia.org/repos/sre/wmf-debci - so if you want to use it, the... [11:56:05] 10serviceops, 10SRE: Migrate conf2* hosts to bullseye - https://phabricator.wikimedia.org/T332010 (10JMeybohm) 05Open→03Resolved Updated etcd-mirror package has been rolled out, resolving this again [12:42:45] 10serviceops, 10Foundational Technology Requests, 10Prod-Kubernetes, 10Kubernetes: Kubernetes v1.23 use PKI for service-account signing (instead of cergen) - https://phabricator.wikimedia.org/T329826 (10JMeybohm) [13:21:59] 10serviceops, 10Fundraising-Backlog, 10Patch-For-Review, 10SecTeam-Processed: FRUP: Add Applepay verification code to donate wiki - https://phabricator.wikimedia.org/T346055 (10akosiaris) 05Open→03Resolved a:03akosiaris https://donate.wikipedia.org/.well-known/apple-developer-merchantid-domain-associ... [13:22:16] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Migrate internal traffic to k8s - https://phabricator.wikimedia.org/T333120 (10Joe) [13:26:42] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, 10Release-Engineering-Team (Seen): Migrate wikifeeds to mw-api-int - https://phabricator.wikimedia.org/T346447 (10Joe) [13:29:35] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, 10Release-Engineering-Team (Seen): Migrate wikifeeds to mw-api-int - https://phabricator.wikimedia.org/T346447 (10Joe) p:05Triage→03Medium [13:31:23] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Migrate internal traffic to k8s - https://phabricator.wikimedia.org/T333120 (10Joe) [13:33:33] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, 10Release-Engineering-Team (Seen): Migrate all eventgate installations to mw-api-int - https://phabricator.wikimedia.org/T346448 (10Joe) [13:33:51] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, 10Release-Engineering-Team (Seen): Migrate all eventgate installations to mw-api-int - https://phabricator.wikimedia.org/T346448 (10Joe) p:05Triage→03Medium a:05Clement_Goubert→03None [13:34:50] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Migrate internal traffic to k8s - https://phabricator.wikimedia.org/T333120 (10Joe) [13:35:04] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, 10Release-Engineering-Team (Seen): Migrate wikifeeds to mw-api-int - https://phabricator.wikimedia.org/T346447 (10Joe) a:05Clement_Goubert→03None [13:44:56] 10serviceops, 10SRE, 10ops-codfw: mw2444 down - https://phabricator.wikimedia.org/T345884 (10Jhancock.wm) 05Open→03Resolved @Clement_Goubert this remained stable for a good while. You should be able to retool it now. Thank you for humoring my wait time. [14:06:38] 10serviceops, 10SRE, 10ops-codfw: mw2444 down - https://phabricator.wikimedia.org/T345884 (10Clement_Goubert) Repooled, thank you! [14:47:26] 10serviceops, 10Fundraising-Backlog, 10SecTeam-Processed: FRUP: Add Applepay verification code to donate wiki - https://phabricator.wikimedia.org/T346055 (10Damilare) Thanks @akosiaris, I can also confirm that the content of the file is now being displayed in that directory. [14:56:55] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install kubernetes10[27-56] - https://phabricator.wikimedia.org/T342533 (10VRiley-WMF) > kubernetes1036 > kubernetes1038 > kubernetes1047 kubernetes1036: Verified that is has correct Serial/Port/CableID. Testing new CableID/Port Old: kubernetes1036... [15:23:29] 10serviceops, 10Observability-Metrics, 10SRE Observability (FY2023/2024-Q1): Identify path forward for k8s deployment of prometheus-statsd-exporter - https://phabricator.wikimedia.org/T343025 (10lmata) @Kappakayala, this is a task for the K8s work. [16:48:16] 10serviceops, 10Release-Engineering-Team, 10Scap: restbase deploys via scap lead to all hosts being disabled in conftool - https://phabricator.wikimedia.org/T346354 (10dancy) @jnuche The last successful restbase deployment was before the `9e49f86 deploy: disable configured `service_name` on secondary target... [17:01:50] 10serviceops, 10Wikidata-Query-Service: Improve concurrency limits configuration of the wdqs updater - https://phabricator.wikimedia.org/T346456 (10dcausse) [17:03:06] 10serviceops, 10Wikidata-Query-Service: Improve concurrency limits configuration of the wdqs updater - https://phabricator.wikimedia.org/T346456 (10dcausse) [17:13:22] 10serviceops, 10Wikidata-Query-Service: Improve concurrency limits configuration of the wdqs updater - https://phabricator.wikimedia.org/T346456 (10bking) [20:58:46] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install kubernetes10[27-56] - https://phabricator.wikimedia.org/T342533 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host kubernetes1036.eqiad.wmnet with OS bullseye [20:58:53] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install kubernetes10[27-56] - https://phabricator.wikimedia.org/T342533 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host kubernetes1036.eqiad.wmnet with OS bullseye executed with errors: - kubernetes10... [20:58:56] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install kubernetes10[27-56] - https://phabricator.wikimedia.org/T342533 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host kubernetes1038.eqiad.wmnet with OS bullseye [20:59:02] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install kubernetes10[27-56] - https://phabricator.wikimedia.org/T342533 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host kubernetes1038.eqiad.wmnet with OS bullseye executed with errors: - kubernetes10... [20:59:32] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install kubernetes10[27-56] - https://phabricator.wikimedia.org/T342533 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host kubernetes1036.eqiad.wmnet with OS bullseye [20:59:38] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install kubernetes10[27-56] - https://phabricator.wikimedia.org/T342533 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host kubernetes1036.eqiad.wmnet with OS bullseye executed with errors: - kubernetes10... [21:03:17] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install kubernetes10[27-56] - https://phabricator.wikimedia.org/T342533 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host kubernetes1047.eqiad.wmnet with OS bullseye [21:03:24] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install kubernetes10[27-56] - https://phabricator.wikimedia.org/T342533 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host kubernetes1047.eqiad.wmnet with OS bullseye executed with errors: - kubernetes10...