[00:00:02] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1059.eqiad.wmnet' (T436113) [00:00:03] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1060.eqiad.wmnet' (T436113) [00:01:21] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) on host 'cloudvirt1060.eqiad.wmnet' (T436113) [00:01:22] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1061.eqiad.wmnet' (T436113) [00:02:33] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet' (T436113) [00:02:35] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1062.eqiad.wmnet' (T436113) [00:08:15] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1062.eqiad.wmnet' (T436113) [00:08:16] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1063.eqiad.wmnet' (T436113) [00:12:46] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1063.eqiad.wmnet' (T436113) [00:12:47] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1064.eqiad.wmnet' (T436113) [00:14:36] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/17 [00:14:39] (03open) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/17 [00:14:55] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/29 [00:14:59] (03open) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/29 [00:15:15] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/24 [00:15:19] (03open) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/24 [00:16:02] (03update) 10raymond-ndibe: Draft: add jobs-api backend to webservice-cli [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/95 (https://phabricator.wikimedia.org/T348755 https://phabricator.wikimedia.org/T431421) [00:16:28] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/56 [00:16:32] (03open) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/56 [00:17:41] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/41 [00:17:48] (03open) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/41 [00:18:13] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1064.eqiad.wmnet' (T436113) [00:18:14] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1065.eqiad.wmnet' (T436113) [00:23:26] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1065.eqiad.wmnet' (T436113) [00:23:27] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1066.eqiad.wmnet' (T436113) [00:28:31] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1066.eqiad.wmnet' (T436113) [00:28:32] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1067.eqiad.wmnet' (T436113) [00:34:23] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1067.eqiad.wmnet' (T436113) [00:34:24] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1068.eqiad.wmnet' (T436113) [00:40:19] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1068.eqiad.wmnet' (T436113) [00:40:20] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1069.eqiad.wmnet' (T436113) [00:45:40] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1069.eqiad.wmnet' (T436113) [00:45:42] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1070.eqiad.wmnet' (T436113) [00:49:06] (03PS1) 10MusikAnimal: SimpleCounter: fix setting of latest edit; add API parameter [labs/xtools] - 10https://gerrit.wikimedia.org/r/1332833 [00:51:21] (03PS2) 10MusikAnimal: SimpleCounter: fix setting of latest edit; add API parameter [labs/xtools] - 10https://gerrit.wikimedia.org/r/1332833 [00:51:22] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1070.eqiad.wmnet' (T436113) [00:51:23] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1071.eqiad.wmnet' (T436113) [00:56:12] (03PS1) 10MusikAnimal: tests: fix various assertions for when APP_IS_WMF is set [labs/xtools] - 10https://gerrit.wikimedia.org/r/1332834 [00:56:32] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1071.eqiad.wmnet' (T436113) [00:56:33] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1072.eqiad.wmnet' (T436113) [01:01:45] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1072.eqiad.wmnet' (T436113) [01:01:46] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1073.eqiad.wmnet' (T436113) [01:01:51] (03PS2) 10MusikAnimal: tests: fix various assertions for when APP_IS_WMF is set [labs/xtools] - 10https://gerrit.wikimedia.org/r/1332834 [01:02:01] (03PS3) 10MusikAnimal: SimpleCounter: fix setting of latest edit; add API parameter [labs/xtools] - 10https://gerrit.wikimedia.org/r/1332833 [01:07:15] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1073.eqiad.wmnet' (T436113) [01:07:16] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1074.eqiad.wmnet' (T436113) [01:12:37] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1074.eqiad.wmnet' (T436113) [01:12:38] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1075.eqiad.wmnet' (T436113) [01:17:53] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1075.eqiad.wmnet' (T436113) [01:17:54] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1076.eqiad.wmnet' (T436113) [01:23:57] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1076.eqiad.wmnet' (T436113) [01:23:58] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1077.eqiad.wmnet' (T436113) [01:27:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:28:21] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1077.eqiad.wmnet' (T436113) [01:28:22] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1078.eqiad.wmnet' (T436113) [01:31:59] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1078.eqiad.wmnet' (T436113) [01:32:00] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1079.eqiad.wmnet' (T436113) [01:36:12] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1079.eqiad.wmnet' (T436113) [01:37:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:42:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:47:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:52:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:08:28] (03close) 10musikanimal: Draft: Initial commit for frontend; work in progress [toolforge-repos/toolviews] - 10https://gitlab.wikimedia.org/toolforge-repos/toolviews/-/merge_requests/8 [02:31:40] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet' (T436113) [02:39:40] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2006-dev.codfw.wmnet' (T436113) [02:54:28] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet' (T436113) [03:04:05] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2006-dev.codfw.wmnet' (T436113) [03:04:17] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet' (T436113) [03:15:04] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2005-dev.codfw.wmnet' (T436113) [03:28:58] 10Tool-wsindex, 06MediaWiki-Media-Platform-Team, 10Wikipedia-iOS-App, 06WikipediaApp, and 2 others: Update apps to support for the move from upload.wikimedia.org to thumb.wikimedia.org - https://phabricator.wikimedia.org/T434821#12273538 (10Bodhisattwa) [03:58:03] (03open) 10bodhisattwa: T434821 [toolforge-repos/wsindex] - 10https://gitlab.wikimedia.org/toolforge-repos/wsindex/-/merge_requests/1 [04:16:38] 10Cloud-VPS (Quota-requests): Quota increase request for project videowiki - https://phabricator.wikimedia.org/T436605 (10Doc_James) 03NEW [04:30:53] 06cloud-services-team, 10Tools, 07Performance Issue: Stalktoy is often slow or not responding - https://phabricator.wikimedia.org/T427941#12273897 (10Pathoschild) This should be fixed now, both for the current level of traffic and with some room to grow. Let me know if you notice any further performance issu... [05:14:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [06:26:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:31:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:33:44] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/36 [06:33:52] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/36 [06:33:58] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/22 [06:34:00] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/22 [06:34:06] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/96 [06:34:23] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/96 [06:34:27] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/189 [06:34:33] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/177 [06:35:00] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/93 [06:35:07] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/43 [06:35:14] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/189 [06:35:16] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/43 [06:35:29] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/37 [06:35:38] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/177 [06:35:46] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/59 [06:35:47] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/97 [06:35:59] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/150 [06:36:09] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/93 [06:36:17] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/121 [06:36:52] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/43 [06:36:56] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/190 [06:36:59] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/374 [06:37:02] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/118 [06:37:20] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/35 [06:37:31] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/52 [06:37:33] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1395 [06:37:40] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/60 [06:37:40] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/43 [06:37:43] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/37 [06:37:45] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/151 [06:37:51] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/78 [06:37:57] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/167 [06:38:02] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/59 [06:38:02] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/105 [06:38:14] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/22 [06:38:23] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/375 [06:38:25] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/94 [06:38:28] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/119 [06:38:31] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/97 [06:38:36] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/341 [06:38:49] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/69 [06:38:55] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/wmcs-k8s-metrics] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/wmcs-k8s-metrics/-/merge_requests/14 [06:38:56] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/150 [06:38:56] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/75 [06:38:57] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/106 [06:39:08] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/76 [06:39:12] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/121 [06:39:17] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/190 [06:39:18] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/374 [06:39:32] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/118 [06:39:36] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/35 [06:39:44] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/52 [06:39:51] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1395 [06:39:51] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/60 [06:39:57] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/151 [06:39:59] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/78 [06:40:01] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/167 [06:40:02] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/105 [06:40:04] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/22 [06:40:05] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/375 [06:40:08] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/94 [06:40:12] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/119 [06:40:16] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/341 [06:40:20] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/69 [06:40:25] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/wmcs-k8s-metrics] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/wmcs-k8s-metrics/-/merge_requests/14 [06:40:28] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/106 [06:40:32] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/75 [06:40:37] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/76 [07:01:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:06:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:17:31] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 3 others: Move cloudvirt1075 from F4 to E4 - https://phabricator.wikimedia.org/T435923#12274066 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by filippo@cumin1003 for hosts: `cloudvirt1075.eqiad.wmn... [07:18:26] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 3 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12274069 (10fgiunchedi) [07:29:53] (03update) 10taavi: Rebrand Elasticsearch alerts as OpenSearch alerts [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/71 (https://phabricator.wikimedia.org/T401818) [07:29:58] (03update) 10taavi: Rebrand Elasticsearch alerts as OpenSearch alerts [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/71 (https://phabricator.wikimedia.org/T401818) [07:30:03] (03update) 10taavi: envvars-admission: Drop old ElasticSearch-related variables [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1394 (https://phabricator.wikimedia.org/T401818) [07:32:58] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12274093 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by filippo@cumin1003 for hosts: `cloudvirt1065... [07:45:19] (03PS1) 10Krinkle: IrcClient: Fix stuck process after connection loss (ping timeout) [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333042 (https://phabricator.wikimedia.org/T327136) [07:45:22] (03PS1) 10Krinkle: IrcClient: Remove `_connected` flag and harden `_registered` flag [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333043 (https://phabricator.wikimedia.org/T327136) [07:48:09] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12274130 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by filippo@cumin1003 for hosts: `cloudvirt1066... [07:51:45] 10Toolforge (Quota-requests): Request increased quota for wiki-economics Toolforge tool - https://phabricator.wikimedia.org/T436614 (10Schiste) 03NEW [08:01:00] (03CR) 10Krinkle: [C:03+2] IrcClient: Fix stuck process after connection loss (ping timeout) [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333042 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [08:01:09] (03CR) 10Krinkle: [C:03+2] IrcClient: Remove `_connected` flag and harden `_registered` flag [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333043 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [08:02:34] (03Merged) 10jenkins-bot: IrcClient: Fix stuck process after connection loss (ping timeout) [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333042 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [08:02:43] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12274181 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by filippo@cumin1003 for hosts: `cloudvirt1067... [08:02:48] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudvirt1072:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [08:02:51] (03Merged) 10jenkins-bot: IrcClient: Remove `_connected` flag and harden `_registered` flag [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333043 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [08:14:18] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudvirt1073:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [08:14:24] 10Toolforge, 06tools-platform-team: Not-in-the-Other-Language fails because PagePileRenderer uses PHP 8.1 syntax on a PHP 7.4 consumer - https://phabricator.wikimedia.org/T436616 (10Narek) 03NEW [08:15:19] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12274233 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by filippo@cumin1003 for hosts: `cloudvirt1072... [08:20:10] (03PS1) 10Krinkle: IrcClient: Increase read timeout from 30s to 200s [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333126 (https://phabricator.wikimedia.org/T327136) [08:20:27] (03CR) 10Krinkle: [C:03+2] IrcClient: Increase read timeout from 30s to 200s [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333126 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [08:21:57] (03Merged) 10jenkins-bot: IrcClient: Increase read timeout from 30s to 200s [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333126 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [08:24:18] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudvirt1074:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [08:32:49] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12274301 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by filippo@cumin1003 for hosts: `cloudvirt1073... [08:44:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:47:23] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12274331 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by filippo@cumin1003 for hosts: `cloudvirt1074... [08:49:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:50:19] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12274333 (10fgiunchedi) a:05fgiunchedi→03VRiley-WMF Hosts have been temporarily decom'd and are ready for you @VRile... [08:50:37] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Move cloudvirt1075 from F4 to E4 - https://phabricator.wikimedia.org/T435923#12274336 (10fgiunchedi) a:05fgiunchedi→03VRiley-WMF Host has been temporarily decom'd and is ready for you @VRiley-WMF [09:00:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [09:01:02] (03approved) 10filippo: Rebrand Elasticsearch alerts as OpenSearch alerts [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/71 (https://phabricator.wikimedia.org/T401818) (owner: 10taavi) [09:05:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [09:13:40] (03open) 10tlepage: refactor: Generalize job event models and push format [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/38 (https://phabricator.wikimedia.org/T432566) [09:14:22] (03update) 10tlepage: refactor: Generalize job event models and push format [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/38 (https://phabricator.wikimedia.org/T432566) [09:14:24] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [09:18:31] (03update) 10tlepage: refactor: Generalize job event models and push format [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/38 (https://phabricator.wikimedia.org/T432566) [09:18:39] (03update) 10tlepage: refactor: Generalize job event models and push format [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/38 (https://phabricator.wikimedia.org/T432566) [09:20:41] (03open) 10tlepage: refactor: Generalize job event models and push format [repos/cloud/toolforge/logs-api] (write-refactor-for-read) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/39 (https://phabricator.wikimedia.org/T432566) [09:24:26] (03open) 10tlepage: events: Add the GET /events endpoint [repos/cloud/toolforge/logs-api] (loki-refactor-for-get-events) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/40 (https://phabricator.wikimedia.org/T432566) [09:25:04] (03update) 10tlepage: refactor: Generalize job event models and push format [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/38 (https://phabricator.wikimedia.org/T432566) [09:25:49] (03update) 10tlepage: refactor: Generalize job event models and push format [repos/cloud/toolforge/logs-api] (write-refactor-for-read) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/39 (https://phabricator.wikimedia.org/T432566) [09:27:37] (03update) 10tlepage: events: Add the GET /events endpoint [repos/cloud/toolforge/logs-api] (loki-refactor-for-get-events) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/40 (https://phabricator.wikimedia.org/T432566) [09:31:38] (03update) 10tlepage: events: Add the GET /events endpoint [repos/cloud/toolforge/logs-api] (loki-refactor-for-get-events) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/40 (https://phabricator.wikimedia.org/T432566) [09:35:34] (03update) 10tlepage: events: Add the GET /events endpoint [repos/cloud/toolforge/logs-api] (loki-refactor-for-get-events) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/40 (https://phabricator.wikimedia.org/T432566) [09:39:19] (03update) 10tlepage: events: Add the GET /events endpoint [repos/cloud/toolforge/logs-api] (loki-refactor-for-get-events) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/40 (https://phabricator.wikimedia.org/T432566) [09:39:24] (03update) 10tlepage: events: Add the GET /events endpoint [repos/cloud/toolforge/logs-api] (loki-refactor-for-get-events) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/40 (https://phabricator.wikimedia.org/T432566) [09:50:20] 10Cloud-VPS (Quota-requests): Quota increase request for project wikispeech - https://phabricator.wikimedia.org/T435967#12274516 (10fgiunchedi) +1 [09:50:28] 10Cloud-VPS (Quota-requests): Quota increase request for project videowiki - https://phabricator.wikimedia.org/T436605#12274518 (10fgiunchedi) +1 [10:31:11] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team: tofu-infra: add cinder volume types - https://phabricator.wikimedia.org/T410148#12274719 (10fnegri) @Mahveotm thanks for your interest in this task! > whether all existing volume types should be brought under tofu-infra; Yes > if the existin... [10:35:17] 10Toolforge, 06tools-platform-team, 07OKR-Work: [loki,alloy,logs-api] reconfigure the labels and add metadata - https://phabricator.wikimedia.org/T432969#12274727 (10TLepage-WMF) `bash # On a Toolforge control plane node, get network access to production Loki via localhost:3100: root@tools-k8s-control-9:~# k... [10:36:04] 10Toolforge, 06tools-platform-team, 07OKR-Work: [loki,alloy,logs-api] reconfigure the labels and add metadata - https://phabricator.wikimedia.org/T432969#12274729 (10TLepage-WMF) [10:36:05] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [logs-api] Implement an endpoint to retrieve system-level event logs - https://phabricator.wikimedia.org/T432566#12274730 (10TLepage-WMF) [10:39:50] 10Toolforge, 06tools-platform-team, 07OKR-Work: [loki,alloy,logs-api] reconfigure the labels and add metadata - https://phabricator.wikimedia.org/T432969#12274743 (10TLepage-WMF) p:05High→03Low Changed priority to "Low" since what remains is only simplification/cleanup. The effort needed to unblock the r... [10:43:04] 10Cloud-VPS (Quota-requests), 10Castor, 10Continuous-Integration-Infrastructure, 06Test Platform (Arusha 30): Provide more disk I/O headroom for the Castor cache volume (integration-castor06) - https://phabricator.wikimedia.org/T435571#12274757 (10fnegri) > tofu-infra defines no Cinder volume types or QoS... [10:49:14] !log fnegri@cloudcumin1001 wikispeech START - Cookbook wmcs.openstack.quota_increase by 3 cores, 6144 ram (T435967) [10:49:18] T435967: Quota increase request for project wikispeech - https://phabricator.wikimedia.org/T435967 [10:49:23] !log fnegri@cloudcumin1001 wikispeech END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 3 cores, 6144 ram (T435967) [10:49:53] 10Cloud-VPS (Quota-requests): Quota increase request for project wikispeech - https://phabricator.wikimedia.org/T435967#12274795 (10fnegri) 05Open→03Resolved a:03fnegri [10:50:07] !log fnegri@cloudcumin1001 videowiki START - Cookbook wmcs.openstack.quota_increase by 4 cores (T436605) [10:50:09] T436605: Quota increase request for project videowiki - https://phabricator.wikimedia.org/T436605 [10:50:15] !log fnegri@cloudcumin1001 videowiki END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 4 cores (T436605) [10:50:27] 10Cloud-VPS (Quota-requests): Quota increase request for project videowiki - https://phabricator.wikimedia.org/T436605#12274801 (10fnegri) 05Open→03Resolved a:03fnegri [11:02:00] 10Toolforge, 06tools-platform-team, 07OKR-Work: [alloy,logs-api] Drop usage of deprecated "job" label - https://phabricator.wikimedia.org/T436627 (10TLepage-WMF) 03NEW [11:02:30] 10Toolforge, 06tools-platform-team, 07OKR-Work: [alloy,logs-api] Drop usage of deprecated "job" label - https://phabricator.wikimedia.org/T436627#12274868 (10TLepage-WMF) [11:02:32] 10Toolforge, 06tools-platform-team, 07Epic, 07OKR-Work: [hypothesis] ST5.4.1 Extend logging capabilities - https://phabricator.wikimedia.org/T432564#12274869 (10TLepage-WMF) [11:03:02] 10Toolforge, 06tools-platform-team, 07OKR-Work: [alloy,logs-api] Drop usage of deprecated "job" label - https://phabricator.wikimedia.org/T436627#12274870 (10TLepage-WMF) [11:03:03] 10Toolforge, 06tools-platform-team, 07OKR-Work: [loki,alloy,logs-api] reconfigure the labels and add metadata - https://phabricator.wikimedia.org/T432969#12274871 (10TLepage-WMF) [11:03:33] 10Toolforge, 06tools-platform-team, 07OKR-Work: [loki,alloy,logs-api] reconfigure the labels and add metadata - https://phabricator.wikimedia.org/T432969#12274872 (10TLepage-WMF) [11:03:35] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [logs-api] Implement an endpoint to retrieve system-level event logs - https://phabricator.wikimedia.org/T432566#12274873 (10TLepage-WMF) [11:04:52] 10Toolforge, 06tools-platform-team, 07OKR-Work: [alloy,logs-api] Drop usage of deprecated "job" label - https://phabricator.wikimedia.org/T436627#12274874 (10TLepage-WMF) p:05Triage→03Low [11:06:34] 10Toolforge, 06tools-platform-team, 07OKR-Work: [loki,alloy,logs-api] reconfigure the labels and add metadata - https://phabricator.wikimedia.org/T432969#12274877 (10TLepage-WMF) 05In progress→03Resolved p:05Low→03High After discussion with @aputhin, I split the remaining effort into T436627 for... [11:39:20] (03Abandoned) 10Jaime Nuche: jobs.yaml: run `update.php` every 15 minutes [labs/tools/train-blockers] - 10https://gerrit.wikimedia.org/r/1055894 (owner: 10Jaime Nuche) [12:03:38] 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team, and 3 others: Move WMCS off of Icinga and introduce alertmanager - https://phabricator.wikimedia.org/T328502#12275161 (10fgiunchedi) [12:04:21] 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team, and 3 others: Move WMCS off of Icinga and introduce alertmanager - https://phabricator.wikimedia.org/T328502#12275163 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi I can confirm that wmcs-owned already are now gone fr... [12:16:46] 06tools-platform-team: jobs-api: support job stop/pause - https://phabricator.wikimedia.org/T436638 (10Raymond_Ndibe) 03NEW [12:24:34] 10Toolforge, 06tools-platform-team: jobs-api: support job stop/pause - https://phabricator.wikimedia.org/T436638#12275274 (10taavi) [12:26:58] 10Toolforge, 06tools-platform-team: [jobs-cli] Use the new `status` fields instead of the old short/long statuses - https://phabricator.wikimedia.org/T430904#12275292 (10Raymond_Ndibe) a:03Raymond_Ndibe [12:55:36] 10Cloud-VPS (Project-requests): Request creation of cepheval VPS project - https://phabricator.wikimedia.org/T436647 (10bking) 03NEW [13:02:17] 10Toolforge, 06tools-infrastructure-team: [toolforge] Rebuild Toolforge workers for bigger node sizes and in new VXLAN network - https://phabricator.wikimedia.org/T390228#12275449 (10fnegri) [13:03:19] 10Cloud-VPS (Project-requests): Request creation of cephvmm VPS project - https://phabricator.wikimedia.org/T436647#12275476 (10bking) [13:04:24] 10Toolforge (Quota-requests): Request increased quota for wiki-economics Toolforge tool - https://phabricator.wikimedia.org/T436614#12275489 (10fnegri) > Per-job memory: 6 GiB → 16 GiB This will not work at the moment, as pods larger than 6GB would fail to schedule. If you need larger pods, you'll have to wait... [13:04:34] 10Cloud-VPS (Quota-requests): Quota increase request for project search - https://phabricator.wikimedia.org/T436242#12275495 (10bking) 05Open→03Invalid @fnegri ACK, it is not related to Search. I have opened T436647 per your request instead. [13:10:53] 10Toolforge (Quota-requests): Request increased quota for wiki-economics Toolforge tool - https://phabricator.wikimedia.org/T436614#12275537 (10Schiste) >>! In T436614#12275489, @fnegri wrote: >> Per-job memory: 6 GiB → 16 GiB > > This will not work at the moment, as pods larger than 6GB would fail to schedule.... [13:14:24] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [13:14:25] FIRING: SystemdUnitFailed: backup_vms.service on cloudbackup1004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:23:59] 06tools-platform-team, 10Toolhub, 13Patch-For-Review: Add banner about Toolhub Evolved - https://phabricator.wikimedia.org/T434556#12275563 (10fnegri) 05In progress→03Resolved The banner is now live. [13:39:05] 10Cloud-VPS (Project-requests), 06Data-Platform-SRE: Request creation of cephvmm VPS project - https://phabricator.wikimedia.org/T436647#12275635 (10Gehel) [13:45:20] (03merge) 10taavi: Rebrand Elasticsearch alerts as OpenSearch alerts [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/71 (https://phabricator.wikimedia.org/T401818) [13:49:25] RESOLVED: SystemdUnitFailed: backup_vms.service on cloudbackup1004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:54:40] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1001.eqiad.wmnet' (T436113) [13:55:15] 10Toolforge, 06tools-platform-team: [jobs-cli] Use the new `status` fields instead of the old short/long statuses - https://phabricator.wikimedia.org/T430904#12275694 (10Raymond_Ndibe) 05Open→03In progress [14:00:38] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1001.eqiad.wmnet' (T436113) [14:00:39] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1002.eqiad.wmnet' (T436113) [14:06:41] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1002.eqiad.wmnet' (T436113) [14:06:42] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1003.eqiad.wmnet' (T436113) [14:13:23] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1003.eqiad.wmnet' (T436113) [14:13:24] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1060.eqiad.wmnet' (T436113) [14:19:23] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1060.eqiad.wmnet' (T436113) [14:19:24] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1061.eqiad.wmnet' (T436113) [14:25:09] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1061.eqiad.wmnet' (T436113) [14:32:16] 10Cloud-VPS (Debian Bullseye Deprecation), 10Tool-bub2: Migrate bub2 away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435761#12275898 (10Andrew) This VM is now shut off. Please follow up on this ticket before restarting. [14:34:09] 10Cloud-VPS (Debian Bullseye Deprecation), 10Capacity Exchange: Migrate capacity-exchange away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435763#12275925 (10Andrew) (the one VM in this project was already shut off as of 9/1) [14:39:10] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12275963 (10Jclark-ctr) @fgiunchedi I am back from sabbatical starting today. I do not see any progress on moving them since your last comment. I will tak... [14:39:27] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12275964 (10Jclark-ctr) a:05fgiunchedi→03Jclark-ctr [14:40:13] 10Cloud-VPS (Project-requests), 06Data-Platform-SRE: Request creation of cephvmm VPS project - https://phabricator.wikimedia.org/T436647#12275966 (10BLiviero-WMF) This request is pretty heavy on our resources, even if temporary. According to https://github.com/wikimedia/operations-puppet/blob/production/mani... [14:40:41] (03open) 10ameisenigel: Sync with production per T263081 [toolforge-repos/wikispore-config] - 10https://gitlab.wikimedia.org/toolforge-repos/wikispore-config/-/merge_requests/6 [14:41:48] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [components-api] Create endpoint to get tools based on a set config - https://phabricator.wikimedia.org/T433690#12275983 (10aputhin) [14:43:56] 10Tools: Not-in-the-Other-Language fails because PagePileRenderer uses PHP 8.1 syntax on a PHP 7.4 consumer - https://phabricator.wikimedia.org/T436616#12275990 (10JJMC89) [15:11:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:16:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:33:26] 06cloud-services-team: Upgrade cloudcumin hosts to Trixie - https://phabricator.wikimedia.org/T436667 (10MoritzMuehlenhoff) 03NEW [15:38:50] 06tools-infrastructure-team: Upgrade cloudcumin hosts to Trixie - https://phabricator.wikimedia.org/T436667#12276313 (10taavi) [15:39:36] 10Cloud-VPS, 06tools-infrastructure-team, 10Maps, 07Upstream: wmf-auto-restart[2162435]: INFO: Could not query the PID(s) of exim4: -11 - https://phabricator.wikimedia.org/T435262#12276320 (10taavi) [15:40:18] 10Cloud-VPS, 06tools-infrastructure-team, 10Maps, 07Upstream: wmf-auto-restart[2162435]: INFO: Could not query the PID(s) of exim4: -11 - https://phabricator.wikimedia.org/T435262#12276325 (10taavi) > Do we really need `wmf-auto-restart` for the systemd-enabled services? Yes, that process is responsible... [15:40:40] 10Cloud-VPS, 06tools-infrastructure-team: cloud-vps: review our CPU overprovisioning policy - https://phabricator.wikimedia.org/T434131#12276326 (10taavi) [15:40:45] 10Cloud-VPS, 06tools-infrastructure-team: Review capacity projection setup for toolforge and cloud-vps - https://phabricator.wikimedia.org/T434128#12276329 (10taavi) [15:41:23] 10PAWS, 06tools-infrastructure-team, 06tools-platform-team: Create alert for https://public-paws.wmcloud.org - https://phabricator.wikimedia.org/T433650#12276331 (10taavi) [15:48:13] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12276376 (10fgiunchedi) Thank you @Jclark-ctr for picking this up! Yes having two hosts in their final allocation one each per E4/F4 would be great. We're... [15:48:50] (03open) 10raymond-ndibe: cli: use new status field instead of status_short/long [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/178 (https://phabricator.wikimedia.org/T430904) [15:50:58] (03update) 10raymond-ndibe: cli: use new status field instead of status_short/long [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/178 (https://phabricator.wikimedia.org/T430904) [15:59:42] 06tools-platform-team, 13Patch-For-Review: components-api: update toolforge models - https://phabricator.wikimedia.org/T436485#12276449 (10aputhin) p:05Triage→03Medium [16:00:26] 10Cloud-VPS, 06tools-infrastructure-team: Fold special maps proxies back to the normal proxy - https://phabricator.wikimedia.org/T431284#12276456 (10taavi) a:03taavi [16:02:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:06:33] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [jobs-cli] Use the new `status` fields instead of the old short/long statuses - https://phabricator.wikimedia.org/T430904#12276488 (10aputhin) p:05Triage→03Medium [16:07:22] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Convert dynamicproxy from nginx to haproxy - https://phabricator.wikimedia.org/T429930#12276495 (10taavi) 05Open→03Resolved This is complete. [16:07:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:07:38] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: web proxy service: limit to one backend per web proxy - https://phabricator.wikimedia.org/T429960#12276498 (10taavi) 05Open→03Resolved [16:08:13] 10Cloud-VPS, 06tools-infrastructure-team: Cleanup multiple backend support from Novaproxy API (& clients) - https://phabricator.wikimedia.org/T436677 (10taavi) 03NEW [16:26:14] !log tools.cluebot3 Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33531533466 (https://github.com/cluebotng/component-configs/commits/348098934d025cf69ee0caa7262b0dfd90ad0b14) [16:26:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot3/SAL [16:27:38] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33531639211 (https://github.com/cluebotng/component-configs/commits/348098934d025cf69ee0caa7262b0dfd90ad0b14) [16:27:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [16:27:44] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33418019250 (https://github.com/cluebotng/component-configs/commits/c689d315eb65ad2cb2f8d0b28293864f7948ce53) [16:27:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [16:56:02] (03update) 10raymond-ndibe: cli: use new status field instead of status_short/long [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/178 (https://phabricator.wikimedia.org/T430904) [17:07:20] 10PAWS, 06tools-platform-team: Upgrade Jupyterhub chart to v4 - https://phabricator.wikimedia.org/T436688 (10taavi) 03NEW [17:10:38] 10Tool-wsindex, 06MediaWiki-Media-Platform-Team, 10Wikipedia-iOS-App, 06WikipediaApp, and 2 others: Update apps to support for the move from upload.wikimedia.org to thumb.wikimedia.org - https://phabricator.wikimedia.org/T434821#12276889 (10Seddon) [17:49:03] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Platform-SRE (2026-08-28 - 2026-09-18), 13Patch-For-Review: Send JSON access logs for dumps.wikimedia.org to Kafka - https://phabricator.wikimedia.org/T425087#12277130 (10Ottomata) I merged the schema promotion and +1ed the EventStream... [17:54:39] (03PS1) 10Krinkle: Switch CVNBot7 and CVNBot8 (cvn-sw) to pycvnbot [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1333269 [17:56:38] (03CR) 10Krinkle: [C:03+2] Switch CVNBot7 and CVNBot8 (cvn-sw) to pycvnbot [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1333269 (owner: 10Krinkle) [17:57:14] (03Merged) 10jenkins-bot: Switch CVNBot7 and CVNBot8 (cvn-sw) to pycvnbot [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1333269 (owner: 10Krinkle) [17:57:23] (03update) 10fnegri: alerts-api: Add initial scaffolding [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T435971) [17:57:31] (03open) 10fnegri: alerts-api: Add initial scaffolding [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T435971) [17:57:58] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-docker-citoid02 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436466#12277168 (10Southparkfan) Note: this instance has been deleted, `deployment-docker-citoid03` exists. Status unknown. [17:58:02] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-docker-cxserver02 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436467#12277169 (10Southparkfan) Note: this instance has been deleted, `deployment-docker-cxserver03` exists. Status unknown. [18:02:19] 10Cloud-VPS (Debian Bullseye Deprecation), 10Countervandalism-Network: Upgrade cvn fleet from Debian 11 Bullseye to Debian 13 Trixie (PHP 7.4 > PHP 8.4) - https://phabricator.wikimedia.org/T395164#12277202 (10Krinkle) [18:02:30] 10Cloud-VPS (Debian Bullseye Deprecation), 10Countervandalism-Network: Upgrade cvn fleet from Debian 11 Bullseye to Debian 13 Trixie (PHP 7.4 > PHP 8.4) - https://phabricator.wikimedia.org/T395164#12277205 (10Krinkle) p:05Triage→03High [18:04:50] (03update) 10raymond-ndibe: cli: use new status field instead of status_short/long [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/178 (https://phabricator.wikimedia.org/T430904) [18:11:23] (03update) 10fnegri: alerts-api: Add initial scaffolding [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T435971) [18:23:31] (03update) 10fnegri: alerts-api: Add initial scaffolding [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T435971) [18:28:31] (03update) 10fnegri: alerts-api: Add initial scaffolding [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T435971) [18:28:32] (03update) 10fnegri: alerts-api: Add initial scaffolding [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T435971) [18:45:36] 10Tool-i18nsearch: duplicate messages when searching without specifying a reop - https://phabricator.wikimedia.org/T436697 (10jsn.sherman) 03NEW [18:46:39] 10Tool-i18nsearch: duplicate messages when searching without specifying a reop - https://phabricator.wikimedia.org/T436697#12277365 (10jsn.sherman) [18:46:45] 10Tool-i18nsearch: duplicate messages when searching without specifying a reop - https://phabricator.wikimedia.org/T436697#12277368 (10jsn.sherman) a:03jsn.sherman [18:48:20] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-webperf2[12] (Bullseye deprecation) - https://phabricator.wikimedia.org/T436464#12277371 (10Southparkfan) On `deployment-webperf22`, Arc Lamp has been broken for a long, long time: `bash Sep 01 18:21:52 deployment-webp... [18:53:25] 10Tool-i18nsearch: duplicate messages when searching without specifying a repo - https://phabricator.wikimedia.org/T436697#12277419 (10jsn.sherman) [19:03:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:19:32] (03PS1) 10Krinkle: apache-config: Include wildcard to ease testing [labs/countervandalism/cvn-infrastructure] - 10https://gerrit.wikimedia.org/r/1333282 (https://phabricator.wikimedia.org/T395164) [19:19:57] (03CR) 10Krinkle: [V:03+2 C:03+2] apache-config: Include wildcard to ease testing [labs/countervandalism/cvn-infrastructure] - 10https://gerrit.wikimedia.org/r/1333282 (https://phabricator.wikimedia.org/T395164) (owner: 10Krinkle) [19:23:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:24:40] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-webperf2[12] (Bullseye deprecation) - https://phabricator.wikimedia.org/T436464#12277567 (10Southparkfan) Given the fact Arc Lamp has been broken for ages anyways, I'm going to follow an adjusted procedure for migratin... [19:31:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:32:36] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12277608 (10Jclark-ctr) @elukey I saw some emails regarding changes to provisioning while I was out on sabbatical. I’m having some issues with provisionin... [19:36:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:40:46] (03merge) 10taavi: puppet_prefix: Handle empty arrays in hiera data [repos/cloud/cloud-vps/tofu-cloudvps] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-cloudvps/-/merge_requests/14 (https://phabricator.wikimedia.org/T436412) (owner: 10bd808) [19:41:38] RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:41:54] (03update) 10taavi: puppet_prefix: Handle empty arrays in hiera data [repos/cloud/cloud-vps/tofu-cloudvps] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-cloudvps/-/merge_requests/14 (https://phabricator.wikimedia.org/T436412) (owner: 10bd808) [19:53:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:53:50] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33551856145 (https://github.com/cluebotng/component-configs/commits/0c6ef4655c137d241bf31075ef088ea0dc85c8a3) [19:53:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [19:54:14] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33551856122 (https://github.com/cluebotng/component-configs/commits/0c6ef4655c137d241bf31075ef088ea0dc85c8a3) [19:54:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [20:03:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:14:46] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/17 [20:14:53] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [20:15:06] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/29 [20:15:12] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [20:15:25] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/24 [20:15:31] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [20:16:29] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/56 [20:16:35] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [20:17:35] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/41 [20:18:10] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [20:18:12] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [21:14:15] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/17 [21:14:16] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [21:14:22] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/29 [21:14:23] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [21:14:29] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/24 [21:14:29] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [21:15:21] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/56 [21:15:22] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [21:16:01] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/41 [21:16:01] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [21:36:47] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Replace deployment-webperf2[12] (Bullseye deprecation) - https://phabricator.wikimedia.org/T436464#12278093 (10Southparkfan) Initial bootstrap of the Wikimedia Performance site fails: ` Notice: /Stage[main]/Httpd/Ht... [21:38:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:02:14] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Replace deployment-webperf2[12] (Bullseye deprecation) - https://phabricator.wikimedia.org/T436464#12278135 (10Southparkfan) Almost done! After applying the `wmf-config/LabsServices.php` changes on `deployment-media... [22:11:52] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Replace deployment-webperf2[12] (Bullseye deprecation) - https://phabricator.wikimedia.org/T436464#12278158 (10Southparkfan) Seems to work fine with `https://webperf.svc.deployment-prep.eqiad1.wikimedia.cloud`. Afte... [22:14:43] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/17 [22:14:55] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [22:15:13] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/29 [22:15:21] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [22:15:38] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/24 [22:15:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:15:46] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [22:16:50] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/56 [22:16:58] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [22:18:07] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/41 [22:18:48] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [22:20:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:25:38] RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:48:41] RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:08:28] (03approved) 10zhaofjx: T436414: Render AI headline links as plain text [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/13 (https://phabricator.wikimedia.org/T436414) (owner: 10chimaifeanyi29) [23:08:53] (03merge) 10zhaofjx: T436414: Render AI headline links as plain text [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/13 (https://phabricator.wikimedia.org/T436414) (owner: 10chimaifeanyi29) [23:09:45] 10Tool-wikinewsie, 13Patch-For-Review: Wikinewsie text should not display external links - https://phabricator.wikimedia.org/T436414#12278281 (10ZhaoFJx) @Chimaifeanyi29 Thank you for your work! [23:10:24] 10Tool-wikinewsie, 13Patch-For-Review: Wikinewsie text should not display external links - https://phabricator.wikimedia.org/T436414#12278282 (10ZhaoFJx) p:05Triage→03High [23:10:37] 10Tool-wikinewsie, 13Patch-For-Review: Wikinewsie text should not display external links - https://phabricator.wikimedia.org/T436414#12278285 (10ZhaoFJx) 05Open→03Resolved [23:12:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:14:25] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/17 [23:14:25] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [23:14:32] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/29 [23:14:33] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [23:14:40] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/24 [23:14:41] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [23:14:41] (03open) 10samwilson: Remove css-sanitizer from composer.local.json [toolforge-repos/wikispore-config] - 10https://gitlab.wikimedia.org/toolforge-repos/wikispore-config/-/merge_requests/7 [23:15:22] (03approved) 10zhaofjx: Fix over-cropping of images [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/8 (https://phabricator.wikimedia.org/T434467) (owner: 10waldir) [23:15:28] (03update) 10zhaofjx: Fix over-cropping of images [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/8 (https://phabricator.wikimedia.org/T434467) (owner: 10waldir) [23:15:36] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/56 [23:15:36] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [23:15:45] (03merge) 10zhaofjx: Fix over-cropping of images [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/8 (https://phabricator.wikimedia.org/T434467) (owner: 10waldir) [23:16:16] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/41 [23:16:17] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [23:17:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:18:12] (03merge) 10samwilson: Remove css-sanitizer from composer.local.json [toolforge-repos/wikispore-config] - 10https://gitlab.wikimedia.org/toolforge-repos/wikispore-config/-/merge_requests/7 [23:19:29] (03update) 10samwilson: Add `import` user group [toolforge-repos/wikispore-config] - 10https://gitlab.wikimedia.org/toolforge-repos/wikispore-config/-/merge_requests/6 (https://phabricator.wikimedia.org/T263081) (owner: 10ameisenigel) [23:22:38] RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:24:29] 10Tool-wikinewsie: Adjustable image alignment on Wikinewsie - https://phabricator.wikimedia.org/T434467#12278305 (10ZhaoFJx) Beyond the default fallback - which is a great patch by @waldyrious, thanks again! - it would be great to also have the manual crop tool suggest by @Pharos as an advanced control method. [23:25:09] (03update) 10zhaofjx: T432938: Add interface language switcher with i18n loading mechanism [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/9 (https://phabricator.wikimedia.org/T432938) (owner: 10chimaifeanyi29) [23:28:43] 10Tool-wikinewsie: Add article summaries to clickthrough modals for the articles below the top article - https://phabricator.wikimedia.org/T436198#12278306 (10ZhaoFJx) p:05Triage→03Medium [23:28:55] 10Tool-wikinewsie: Add article summaries to clickthrough modals for the articles below the top article - https://phabricator.wikimedia.org/T436198#12278310 (10ZhaoFJx) 05Open→03In progress [23:29:59] 10Tool-wikinewsie: Auto-Deploy Wikinewsie Commits - https://phabricator.wikimedia.org/T435735#12278313 (10ZhaoFJx) 05Stalled→03In progress [23:30:37] 10Tool-wikinewsie: Wikinewsie aggregation of stories - https://phabricator.wikimedia.org/T436544#12278314 (10ZhaoFJx) p:05Triage→03Medium [23:31:38] 10Tool-wikinewsie: Wikinewsie suggested headline from AI recap - https://phabricator.wikimedia.org/T436419#12278319 (10ZhaoFJx) p:05Triage→03High [23:32:18] 10Tool-wikinewsie, 10LPL sprints, 06translatewiki.net, 10LPL Projects (Ongoing maintenance): Add Wikinewsie to translatewiki.net - https://phabricator.wikimedia.org/T435739#12278324 (10ZhaoFJx) 05Open→03In progress [23:33:13] 10Tool-wikinewsie, 10LPL sprints, 06translatewiki.net, 10LPL Projects (Ongoing maintenance): Add Wikinewsie to translatewiki.net - https://phabricator.wikimedia.org/T435739#12278327 (10ZhaoFJx) 05In progress→03Open [23:35:40] 06tools-platform-team: jobs-api: job with failing healthcheck should not be marked running in the new status implementation - https://phabricator.wikimedia.org/T436726 (10Raymond_Ndibe) 03NEW [23:37:26] (03open) 10raymond-ndibe: status: job with failing healthcheck should not be marked running [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/376 (https://phabricator.wikimedia.org/T436726) [23:38:41] (03update) 10raymond-ndibe: cli: use new status field instead of status_short/long [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/178 (https://phabricator.wikimedia.org/T430904) [23:50:06] (03update) 10raymond-ndibe: status: job with failing healthcheck should not be marked running [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/376 (https://phabricator.wikimedia.org/T436726) [23:59:57] (03update) 10raymond-ndibe: cli: use new status field instead of status_short/long [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/178 (https://phabricator.wikimedia.org/T430904)