[00:02:18] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate text-to-speech away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435785#12263861 (10TheresNoTime) [00:03:14] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate wmcs-uptime away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435759#12263862 (10TheresNoTime) [00:46:14] 10Toolforge, 06tools-platform-team: `toolforge jobs dump` produces `filelog: 'yes'` which `toolforge jobs load` says is deprecated syntax - https://phabricator.wikimedia.org/T436314 (10bd808) 03NEW [02:07:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:17:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:26:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [03:29:56] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate discourse away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T436152#12264056 (10Samwilson) Actually, I've found a newer one: `wikimedia-discourse-2019-05-24-033619-v20190110201340.tar.gz` I don't know if I can make the dumps I've... [03:31:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [03:44:26] 10Tool-Gerrit-Patch-Uploader: "Project" dropdown is empty, preventing patch submission - https://phabricator.wikimedia.org/T436321 (10Od1n) 03NEW [03:46:11] 10Tool-Gerrit-Patch-Uploader: "Project" dropdown is empty, preventing patch submission - https://phabricator.wikimedia.org/T436321#12264077 (10Od1n) p:05Triage→03High [04:09:34] (03close) 10legoktm: Updates (July 2026) [toolforge-repos/slf-bot] - 10https://gitlab.wikimedia.org/toolforge-repos/slf-bot/-/merge_requests/3 [04:15:32] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate spi-tools away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435782#12264087 (10komla) @RoySmith thanks! I will close this task as resolved. [04:15:48] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate spi-tools away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435782#12264088 (10komla) 05Open→03Resolved a:03komla [04:32:11] (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/15 [04:32:25] (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/27 [04:32:42] (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/22 [04:33:47] (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/54 [04:34:51] (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/39 [04:38:00] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate webperf away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435788#12264102 (10Aklapper) @komla: Nobody who can do anything will see this ticket when no active project tags or subscribers are added [04:44:00] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate spi-tools away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435782#12264126 (10JJMC89) a:05komla→03RoySmith [04:49:25] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate webperf away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435788#12264131 (10komla) @Aklapper yeah, this was a curious one. When I was adding the comment earlier, I thought the subscribers I added had unsubscribed themselves! So... [04:53:13] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate webperf away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435788#12264133 (10komla) @aaron @Krinkle @tstarling this is a reminder that the `webperf` project will be shut down on Tuesday, September 1st. Reach out if you cannot me... [05:00:08] 10Tool-gitlab-account-approval: Automatically approve GitLab accounts for users with editinterface in any wiki - https://phabricator.wikimedia.org/T436323 (10Bugreporter) 03NEW [05:14:17] (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/15 [05:14:24] (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/27 [05:14:32] (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/22 [05:15:26] (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/54 [05:16:07] (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/39 [05:32:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:37:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:08:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:31:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:36:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:47:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:52:38] RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:07:29] 10Tool-gitlab-account-approval: Automatically approve GitLab accounts for users with editinterface in any wiki - https://phabricator.wikimedia.org/T436323#12264295 (10Aklapper) > But GitLab account is not by default approved. For the records, this describes the current situation: {F65766561,size=full} [08:30:25] (03open) 10samwilson: Deprecate available_langs API, and use attributes for Open API details [toolforge-repos/ocr] - 10https://gitlab.wikimedia.org/toolforge-repos/ocr/-/merge_requests/17 (https://phabricator.wikimedia.org/T330061) [09:22:44] (03approved) 10filippo: Add a new 'legacy' flavor to replace one that was accidentally removed [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/346 (https://phabricator.wikimedia.org/T436275) (owner: 10andrew) [09:29:58] 06tools-infrastructure-team, 13Patch-For-Review: Track toolforge virtualization footprint: ram/cores/instance &c - https://phabricator.wikimedia.org/T436275#12264672 (10fgiunchedi) I like the prometheus idea and I think we should go for that. My preference is to keep/export said counts on the control plane sid... [09:37:23] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1065' (T435921) [09:37:30] T435921: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921 [09:41:07] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.set_maintenance [09:41:10] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0) [09:41:19] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.set_maintenance [09:41:22] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0) [09:41:29] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.set_maintenance [09:41:32] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0) [09:41:35] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.set_maintenance [09:41:38] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0) [09:41:48] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.set_maintenance [09:41:51] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0) [09:41:58] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.set_maintenance [09:42:01] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0) [09:49:43] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1065' (T435921) [09:49:51] T435921: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921 [10:04:13] 06tools-platform-team, 06Abstract Wikipedia team, 06Collaboration-Services, 06Content-Platform-Team, and 6 others: Upgrade Envoy to v1.39.0 - https://phabricator.wikimedia.org/T421418#12264839 (10Blake) [10:08:56] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:33:36] (03CR) 10FNegri: [C:03+1] "LGTM" [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1310212 (https://phabricator.wikimedia.org/T432078) (owner: 10Raymond Ndibe) [10:44:29] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1066' (T435921) [10:44:37] T435921: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921 [10:58:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:59:44] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1066' (T435921) [10:59:51] T435921: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921 [11:03:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:13:52] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1067' (T435921) [11:14:00] T435921: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921 [11:23:23] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1067' (T435921) [11:23:30] T435921: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921 [12:18:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:31:16] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1072' (T435921) [12:31:23] T435921: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921 [12:33:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:35:57] 10Cloud-VPS (Quota-requests): Quota increase request for project search - https://phabricator.wikimedia.org/T436242#12265282 (10bking) [12:38:38] RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:44:33] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1072' (T435921) [12:44:40] T435921: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921 [12:47:44] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1073' (T435921) [12:57:40] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1073' (T435921) [12:57:47] T435921: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921 [12:59:07] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Upgrade clouddumps hosts to bookworm/trixie - https://phabricator.wikimedia.org/T403154#12265397 (10Gehel) [12:59:22] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Platform-SRE (2026-08-28 - 2026-09-18), 13Patch-For-Review: Send JSON access logs for dumps.wikimedia.org to Kafka - https://phabricator.wikimedia.org/T425087#12265401 (10Gehel) [13:00:30] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Engineering, and 4 others: Create wiki replicas views for globaljsonlinks tables - https://phabricator.wikimedia.org/T387419#12265431 (10Gehel) [13:00:36] 10Cloud Services Proposals, 10Data-Services, 06tools-platform-team, 06Data-Persistence, and 2 others: Decision request - Who runs wikireplicas cookbooks - https://phabricator.wikimedia.org/T382607#12265435 (10Gehel) [13:00:56] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Engineering, and 4 others: Set up x1 replication to Wiki Replicas - https://phabricator.wikimedia.org/T395881#12265439 (10Gehel) [13:01:36] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1074' (T435921) [13:07:17] 06tools-platform-team: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12265574 (10TLepage-WMF) For future similar tasks, one thing was missing in the checklist above: membership in https://gitlab.wikimedia.org/groups/toolforge-repos/... [13:08:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:11:48] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1074' (T435921) [13:11:55] T435921: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921 [13:12:59] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1075' (T435921) [13:13:00] !log filippo@cloudcumin1001 admin END (ERROR) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=97) on host 'cloudvirt1075' (T435921) [13:13:09] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1075' (T435923) [13:13:15] T435923: Move cloudvirt1075 from F4 to E4 - https://phabricator.wikimedia.org/T435923 [13:13:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:18:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:19:30] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1075' (T435923) [13:19:37] T435923: Move cloudvirt1075 from F4 to E4 - https://phabricator.wikimedia.org/T435923 [13:23:30] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 10BetaFeatures, and 6 others: Create view for betafeatures_user_counts table in wiki replicas - https://phabricator.wikimedia.org/T402145#12265711 (10Gehel) [13:32:05] 10Cloud-VPS, 06tools-infrastructure-team, 10Cumin, 06Infrastructure-Foundations: Enable cumin hostfile backend on cloudcumin hosts - https://phabricator.wikimedia.org/T433916#12265790 (10elukey) @Andrew o/ is it a duplicate of T380789#10363343 ? [13:39:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:41:00] 06tools-platform-team, 06Abstract Wikipedia team, 06Collaboration-Services, 06Content-Platform-Team, and 6 others: Upgrade Envoy to v1.39.0 - https://phabricator.wikimedia.org/T421418#12265834 (10Gehel) [13:44:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:56:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:01:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:04:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:08:56] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:09:38] RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:27:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:29:01] 10Toolforge: Toolforge Builds API returns 503 because its Harbor authentication is unauthorized - https://phabricator.wikimedia.org/T436244#12266009 (10aputhin) [14:29:52] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: Toolforge: drop PodSecurityPolicy - https://phabricator.wikimedia.org/T368142#12266011 (10aputhin) [14:30:06] 10Data-Services: Allow querying for global edit count - https://phabricator.wikimedia.org/T435728#12266016 (10aputhin) [14:30:47] 10Quarry: Define a robot policy for Quarry then create robots.txt file - https://phabricator.wikimedia.org/T224338#12266021 (10aputhin) [14:32:10] 10Tool-paulina, 07Kubernetes: Toolforge: paulina.toolforge.org Python/Flask tool experiencing connection errors - https://phabricator.wikimedia.org/T425784#12266025 (10aputhin) [14:32:12] 10Toolforge: [k8s, kube-proxy] "udpIdleTimeout" KubeProxyConfiguration deprecation - https://phabricator.wikimedia.org/T373537#12266026 (10aputhin) [14:33:36] 06cloud-services-team, 10Toolforge: !log automated deployments so that a tool’s SAL records system changes - https://phabricator.wikimedia.org/T401963#12266028 (10aputhin) [14:34:36] 10PAWS, 06tools-infrastructure-team: Make sure dumps-nfs mounts are propagated inside PAWS containers - https://phabricator.wikimedia.org/T432583#12266030 (10aputhin) [14:35:14] 06cloud-services-team, 10Tool-paulina, 10Toolforge: Automated traffic affects paulina.toolforge.org performance - https://phabricator.wikimedia.org/T432878#12266031 (10aputhin) [14:37:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:40:41] 06cloud-services-team, 10Toolforge, 06tools-infrastructure-team: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.33 - https://phabricator.wikimedia.org/T408785#12266074 (10fnegri) [14:48:41] RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:08:20] (03open) 10sarai-wmf: Update reuse scenarios across attribution signals [toolforge-repos/wikimedia-attribution] - 10https://gitlab.wikimedia.org/toolforge-repos/wikimedia-attribution/-/merge_requests/5 [15:08:24] (03merge) 10sarai-wmf: Update reuse scenarios across attribution signals [toolforge-repos/wikimedia-attribution] - 10https://gitlab.wikimedia.org/toolforge-repos/wikimedia-attribution/-/merge_requests/5 [15:09:28] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [15:10:01] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [15:10:27] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/346 [15:11:05] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/346 [15:13:37] (03merge) 10andrew: Add a new 'legacy' flavor to replace one that was accidentally removed [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/346 (https://phabricator.wikimedia.org/T436275) [15:13:47] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [15:15:01] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [15:16:27] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for main branch [15:16:59] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch [15:38:33] 06tools-infrastructure-team: Track toolforge virtualization footprint: ram/cores/instance &c - https://phabricator.wikimedia.org/T436275#12266302 (10Andrew) Will prometheus get crabby if I only write the metrics once per day? The queries will be a bit expensive and it's not like project size changes minute-by-mi... [15:40:09] 10Cloud-VPS, 06tools-infrastructure-team, 10Cumin, 06Infrastructure-Foundations: Enable cumin hostfile backend on cloudcumin hosts - https://phabricator.wikimedia.org/T433916#12266303 (10Andrew) >>! In T433916#12265790, @elukey wrote: > @Andrew o/ is it a duplicate of T380789#10363343 ? Kind of! That task... [15:54:30] 10Tool-gitlab-account-approval: Automatically approve GitLab accounts for users with editinterface in any wiki - https://phabricator.wikimedia.org/T436323#12266342 (10bd808) > group that grants editinterface (e.g. sysops and interface admins). If it is, the account should be automatically approved. I like the... [16:12:02] 10Tool-gitlab-account-approval: Automatically approve GitLab accounts for users with editinterface in any wiki - https://phabricator.wikimedia.org/T436323#12266372 (10Ammarpad) >>! In T436323#12266342, @bd808 wrote: > Should all project wikis be treated equally for something like this? Like is it roughly the sam... [16:12:18] (03merge) 10countcount: Update pnpm to v11.24.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/15 (owner: 10renovatebot) [16:12:25] (03merge) 10countcount: Update pnpm to v11.24.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/27 (owner: 10renovatebot) [16:12:29] (03merge) 10countcount: Update pnpm to v11.24.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/22 (owner: 10renovatebot) [16:12:35] (03merge) 10countcount: Update pnpm to v11.24.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/54 (owner: 10renovatebot) [16:12:42] (03merge) 10countcount: Update pnpm to v11.24.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/39 (owner: 10renovatebot) [16:16:43] (03update) 10renovatebot: Update cimg/rust Docker tag to v1.98 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/36 [17:14:46] 10Tool-wikinewsie, 13Patch-For-Review: Add tooltip explanations for "Mix", "Editors", "Edits", etc - https://phabricator.wikimedia.org/T435732#12266547 (10Chimaifeanyi29) a:03Chimaifeanyi29 [17:15:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:20:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:10:47] (03open) 10countcount: Fix nightly Clippy diagnostics [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/27 [19:13:58] (03open) 10countcount: Fix Clippy on the current nightly [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/79 [19:21:17] 06cloud-services-team, 10Toolforge, 06tools-platform-team: Install mariadb-dump on Toolforge bastions - https://phabricator.wikimedia.org/T378882#12266926 (10LucasWerkmeister) I finally thought of a better way to work around this: port-forward MariaDB through SSH and then run `mariadb-dump` locally (after al... [19:24:31] (03update) 10countcount: Fix nightly Clippy diagnostics [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/27 [19:24:38] (03update) 10countcount: Fix nightly Clippy and dependency audit failures [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/27 [19:26:11] (03update) 10countcount: Fix Clippy on the current nightly [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/79 [19:26:19] (03update) 10countcount: Fix Clippy and dependency audit checks [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/79 [21:48:22] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 07Epic, 06Release-Engineering-Team (Priority Backlog 📥): Migrate deployment-prep away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T401839#12267376 (10Southparkfan) [22:22:57] 06tools-infrastructure-team: [tofu-cloudvps] cloudvps_puppet_prefix.hiera is not idempotent and should be - https://phabricator.wikimedia.org/T436412 (10Andrew) 03NEW [22:23:44] 10Cloud-VPS (Debian Bullseye Deprecation), 06tools-infrastructure-team: [tofu-cloudvps] cloudvps_puppet_prefix.hiera is not idempotent and should be - https://phabricator.wikimedia.org/T436412#12267454 (10Andrew) [22:24:55] 10Cloud-VPS (Debian Bullseye Deprecation), 06tools-infrastructure-team: [tofu-cloudvps] cloudvps_puppet_prefix.hiera is not idempotent and should be - https://phabricator.wikimedia.org/T436412#12267455 (10Andrew) I am pretty sure that right before we [[ https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-c... [22:28:40] 10Cloud-VPS (Debian Bullseye Deprecation), 06tools-infrastructure-team: [tofu-cloudvps] cloudvps_puppet_prefix.hiera is not idempotent and should be - https://phabricator.wikimedia.org/T436412#12267460 (10bd808) If the "oops there is already a prefix here" step failed with an error message that gave the upstre... [23:03:09] 06tools-infrastructure-team, 10wikitech.wikimedia.org: wikitech-static: get better at cleaning up old containers and/or establish a cadence for rebuilding the host - https://phabricator.wikimedia.org/T432391#12267498 (10Andrew) Let's see what's happening! I have two VMs now, one of which is about an hour old a... [23:25:41] 06tools-infrastructure-team, 10wikitech.wikimedia.org, 13Patch-For-Review: wikitech-static: get better at cleaning up old containers and/or establish a cadence for rebuilding the host - https://phabricator.wikimedia.org/T432391#12267510 (10Andrew) I've built a fresh VM with logrotate. will check on the growt... [23:48:28] (03update) 10pacmand: Add optional image pre-processing to suppress show-through [toolforge-repos/ocr] - 10https://gitlab.wikimedia.org/toolforge-repos/ocr/-/merge_requests/14 (https://phabricator.wikimedia.org/T435428) [23:53:50] (03update) 10pacmand: Add optional image pre-processing to suppress show-through [toolforge-repos/ocr] - 10https://gitlab.wikimedia.org/toolforge-repos/ocr/-/merge_requests/14