[00:38:28] <wikibugs>	 (03PS3) 10BryanDavis: labsauth: Log a LabsUser's name and dn when LdapUser fetch fails [labs/striker] - 10https://gerrit.wikimedia.org/r/1324800 (https://phabricator.wikimedia.org/T434684)
[00:50:40] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team (Doing 😎): Planning for a plan to migrate Beta Cluster again - https://phabricator.wikimedia.org/T433998#12210303 (10bd808) Update: Southparkfan won't be available for a bit, but he and I plan to sync up in t...
[01:17:49] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.32.13 to 1.33.13 (TT433131)
[01:20:54] <wmcs-alerts>	 RESOLVED: ToolforgeKubernetesNodeNotReady: Kubernetes node toolsbeta-test-k8s-control-10 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady
[01:23:15] <wmcs-alerts>	 FIRING: [2x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server toolsbeta-test-k8s-control-11.toolsbeta.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown
[01:32:24] <wmcs-alerts>	 FIRING: ToolforgeKubernetesNodeNotReady: Kubernetes node toolsbeta-test-k8s-control-11 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady
[01:42:18] <logmsgbot_cloud>	 andrew@cloudcumin1001 upgrade (PID 3109113) is awaiting input
[01:42:24] <wmcs-alerts>	 RESOLVED: ToolforgeKubernetesNodeNotReady: Kubernetes node toolsbeta-test-k8s-control-11 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady
[01:43:15] <wmcs-alerts>	 RESOLVED: [2x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server toolsbeta-test-k8s-control-11.toolsbeta.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDo
[01:47:50] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-11 from 1.32.13 to 1.33.13 (TT433131)
[01:48:02] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131)
[01:48:02] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131)
[01:49:22] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-13
[01:49:22] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-13
[01:50:22] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-13
[01:51:52] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-13
[01:56:34] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 (T433131)
[01:56:38] <stashbot>	 T433131: Upgrade toolsbeta to Kubernetes version 1.33 - https://phabricator.wikimedia.org/T433131
[01:57:39] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 (T433131)
[01:58:15] <wmcs-alerts>	 FIRING: ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server toolsbeta-test-k8s-control-13.toolsbeta.eqiad1.wikimedia.cloud is MAINT - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown
[01:58:37] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 (T433131)
[01:59:38] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 (T433131)
[02:01:03] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-gateway-3 from 1.32.13 to 1.33.13 (TT433131)
[02:01:55] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-gateway-3 from 1.32.13 to 1.33.13 (TT433131)
[02:01:56] <wikibugs>	 06cloud-services-team, 10Toolforge, 06tools-infrastructure-team: Upgrade toolsbeta to Kubernetes version 1.33 - https://phabricator.wikimedia.org/T433131#12210401 (10Andrew) [] no mention in the upgrade guide of gateway nodes
[02:02:05] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-gateway-2 from 1.32.13 to 1.33.13 (TT433131)
[02:02:59] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-gateway-2 from 1.32.13 to 1.33.13 (TT433131)
[02:03:09] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-gateway-1 from 1.32.13 to 1.33.13 (TT433131)
[02:04:03] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-gateway-1 from 1.32.13 to 1.33.13 (TT433131)
[02:08:14] <wmcs-alerts>	 RESOLVED: ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server toolsbeta-test-k8s-control-13.toolsbeta.eqiad1.wikimedia.cloud is MAINT - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown
[02:09:48] <wikibugs>	 (03update) 10raymond-ndibe: test continuous job webservice [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1131 (https://phabricator.wikimedia.org/T348755)
[02:16:20] <wikibugs>	 (03PS1) 10Krinkle: Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136)
[02:21:08] <wikibugs>	 (03CR) 10Krinkle: "I'll be reviewing this in depth, and runnning a bunch of tests." [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle)
[03:39:44] <wikibugs>	 (03PS2) 10Krinkle: [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136)
[03:40:16] <wikibugs>	 (03CR) 10CI reject: [V:04-1] [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle)
[03:41:39] <wikibugs>	 (03PS3) 10Krinkle: [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136)
[03:42:09] <wikibugs>	 (03CR) 10CI reject: [V:04-1] [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle)
[04:14:14] <wikibugs>	 (03update) 10renovatebot: Update dependency @napi-rs/wasm-runtime to v1.2.2 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/5
[04:14:27] <wikibugs>	 (03update) 10renovatebot: Update dependency @napi-rs/wasm-runtime to v1.2.3 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/5
[05:13:59] <wikibugs>	 (03update) 10renovatebot: Update dependency @napi-rs/wasm-runtime to v1.2.3 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/5
[05:20:24] <wikibugs>	 10Tool-linkdata: Make the HTTP client a first-class citizen - https://phabricator.wikimedia.org/T434717#12210570 (10Audiodude) a:03Audiodude
[05:31:42] <wikibugs>	 10Tool-linkdata: Make the HTTP client a first-class citizen - https://phabricator.wikimedia.org/T434717#12210575 (10Audiodude) 05Open→03Resolved Shipped in [[ https://gitlab.wikimedia.org/tmoney/linkdata/-/commit/676f58c | 676f58c ]] — all four gaps:  - **CORS** on `/datasets/`: `Access-Control-Allow-Ori...
[05:48:20] <wikibugs>	 (03open) 10countcount: Replace hand-rolled PHP deserialization with phpserz [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/76
[06:11:29] <wikibugs>	 (03update) 10countcount: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/29 (owner: 10renovatebot)
[06:11:36] <wikibugs>	 (03update) 10countcount: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/46 (owner: 10renovatebot)
[06:11:39] <wikibugs>	 (03update) 10countcount: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/14 (owner: 10renovatebot)
[06:11:42] <wikibugs>	 (03update) 10countcount: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/19 (owner: 10renovatebot)
[06:11:47] <wikibugs>	 (03update) 10countcount: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/6 (owner: 10renovatebot)
[06:14:28] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/6
[06:14:29] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/6
[06:14:44] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/19
[06:14:46] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/19
[06:15:02] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/14
[06:15:03] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/14
[06:16:07] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/46
[06:16:12] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/46
[06:17:52] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/29
[06:17:54] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/29
[06:20:44] <wikibugs>	 10Tool-clip2commons, 13Patch-For-Review: Remember previously selected license - https://phabricator.wikimedia.org/T434084#12210606 (10Dev) Please check new note [[ https://gitlab.wikimedia.org/toolforge-repos/clip2commons/-/merge_requests/11#note_235327 | Merge Request !11 ]]
[06:25:32] <wikibugs>	 (03PS4) 10Krinkle: [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136)
[06:26:04] <wikibugs>	 (03CR) 10CI reject: [V:04-1] [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle)
[07:07:25] <wikibugs>	 06tools-infrastructure-team, 06Data-Platform-SRE, 06Wikidata Platform Team, 13Patch-For-Review: Migrate to dumps-nfs.w.o in production - https://phabricator.wikimedia.org/T432212#12210704 (10Gehel) Tagging #wikidata_platform_team for visibility. We can probably completely remove that mount for w[cd]qs* hos...
[07:08:45] <wikibugs>	 06tools-infrastructure-team, 06Wikidata Platform Team, 06Data-Platform-SRE (2026-08-07 - 2026-08-28), 13Patch-For-Review: Migrate to dumps-nfs.w.o in production - https://phabricator.wikimedia.org/T432212#12210706 (10Gehel)
[07:13:58] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/6
[07:14:06] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/19
[07:14:11] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/14
[07:15:04] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/46
[07:15:41] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/29
[07:26:21] <wikibugs>	 06tools-infrastructure-team, 06Wikidata Platform Team, 06Data-Platform-SRE (2026-08-07 - 2026-08-28), 13Patch-For-Review: Migrate to dumps-nfs.w.o in production - https://phabricator.wikimedia.org/T432212#12210764 (10fgiunchedi) >>! In T432212#12210704, @Gehel wrote: > Tagging #wikidata_platform_team for v...
[07:29:57] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1049' (T431682)
[07:30:02] <stashbot>	 T431682: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682
[07:38:04] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1049' (T431682)
[07:38:10] <stashbot>	 T431682: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682
[07:38:42] <wikibugs>	 10Tools: Zoom viewer URL is not available - https://phabricator.wikimedia.org/T434745#12210806 (10Aklapper) Thanks for reporting this. See https://commons.wikimedia.org/wiki/Help:Gadget-ZoomViewer and https://phabricator.wikimedia.org/T428524#12003358
[07:38:53] <wikibugs>	 10Tools: Zoom viewer URL is not available - https://phabricator.wikimedia.org/T434745#12210811 (10Aklapper) →14Duplicate dup:03T428524
[07:38:56] <wikibugs>	 10Tools, 06Commons: Zoomviewer not working and thumbs of big images - https://phabricator.wikimedia.org/T428524#12210813 (10Aklapper)
[07:39:50] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1051' (T431682)
[07:50:12] <wikibugs>	 (03CR) 10David Caro: [C:03+1] "LGTM" [labs/striker] - 10https://gerrit.wikimedia.org/r/1324800 (https://phabricator.wikimedia.org/T434684) (owner: 10BryanDavis)
[07:53:36] <wikibugs>	 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12210883 (10fgiunchedi) >>! In T431682#12209331, @VRiley-WMF wrote: > Hey @fgiunchedi  Yes, that would work for me! Woul...
[07:54:45] <wikibugs>	 (03merge) 10filippo: Update tests with /mnt/nfs/dumps directory [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/98 (https://phabricator.wikimedia.org/T411248)
[07:56:27] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1051' (T431682)
[07:56:33] <stashbot>	 T431682: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682
[07:58:50] <wikibugs>	 06cloud-services-team, 10Toolforge, 06tools-infrastructure-team: Upgrade toolsbeta to Kubernetes version 1.33 - https://phabricator.wikimedia.org/T433131#12210896 (10dcaro) there's no control-13 in the cluster that I can see: ` root@toolsbeta-test-k8s-control-12:~# kubectl get nodes NAME...
[07:59:31] <wikibugs>	 10PAWS, 06tools-infrastructure-team, 06tools-platform-team: Make sure dumps-nfs mounts are propagated inside PAWS containers - https://phabricator.wikimedia.org/T432583#12210900 (10fgiunchedi) I realized github pull requests won't show up here. I got these out:  https://github.com/toolforge/paws/pull/521 htt...
[08:01:32] <wikibugs>	 06cloud-services-team, 10Toolforge, 06tools-infrastructure-team: Upgrade toolsbeta to Kubernetes version 1.33 - https://phabricator.wikimedia.org/T433131#12210903 (10dcaro) Oh, it seems control 13 was removed: ` !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_c...
[08:02:00] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1054' (T431682)
[08:02:08] <stashbot>	 T431682: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682
[08:08:53] <wikibugs>	 06cloud-services-team, 10Toolforge, 06tools-infrastructure-team: Upgrade toolsbeta to Kubernetes version 1.33 - https://phabricator.wikimedia.org/T433131#12210915 (10dcaro) > anyhow, looking into the failed probes.  For some reason the admin tool webservice was failing trying to run `whoami`, restarted using...
[08:09:29] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli
[08:12:51] <wikibugs>	 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: openstack: alert for cloudvirts without aggregate or with unexpected set of them - https://phabricator.wikimedia.org/T284747#12210925 (10fgiunchedi) The patch at https://gerrit.wikimedia.org/r/c/operations/puppet/+/1313958 will export compute serv...
[08:19:48] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1054' (T431682)
[08:19:57] <stashbot>	 T431682: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682
[08:21:08] <wmcs-alerts>	 RESOLVED: [4x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[08:21:54] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-cli
[08:22:12] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1055' (T431682)
[08:28:02] <wikibugs>	 (03update) 10countcount: Replace hand-rolled PHP deserialization with phpserz [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/76
[08:28:12] <wikibugs>	 (03update) 10countcount: Replace hand-rolled PHP deserialization with phpserz [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/76
[08:42:41] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1055' (T431682)
[08:42:51] <stashbot>	 T431682: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682
[08:56:35] <wikibugs>	 10Tool-itwiki, 07Upstream: Aggiungere un'etichetta alle modifiche di JWB - https://phabricator.wikimedia.org/T363923#12211161 (10valerio.bozzolan) 🤩
[08:58:56] <wikibugs>	 10Tool-itwiki: BotCancellazioni: rollback "cllimit=500" workaround after T433922 has been resolved - https://phabricator.wikimedia.org/T434598#12211177 (10valerio.bozzolan) > The workround is also a performance optimization  Well, this may be a performance optimization for the bot, but it's a trade-off decreasin...
[09:00:41] <wikibugs>	 (03update) 10dcaro: tests: add behavior tests for previously untested code branches [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/362 (https://phabricator.wikimedia.org/T434166) (owner: 10raymond-ndibe)
[09:01:38] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1056' (T431682)
[09:01:43] <stashbot>	 T431682: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682
[09:02:15] <wikibugs>	 (03update) 10dcaro: refactor: move observability fields out of CommonOptions into ObservabilityOptions [repos/cloud/toolforge/jobs-api] (rename_common_job_to_common_options) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/363 (https://phabricator.wikimedia.org/T434166) (owner: 10raymond-ndibe)
[09:02:57] <wikibugs>	 10Tool-itwiki: BotCancellazioni: rollback "cllimit=500" workaround after T433922 has been resolved - https://phabricator.wikimedia.org/T434598#12211196 (10valerio.bozzolan) I shared the above comment in the assumption that current server limits were selected to consume less resources, but if I'm wrong and if som...
[09:06:11] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli
[09:15:18] <wikibugs>	 (03update) 10dcaro: models: move cmd field out of CommonJob into job-type models [repos/cloud/toolforge/jobs-api] (increase_test_coverage) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/364 (https://phabricator.wikimedia.org/T434166) (owner: 10raymond-ndibe)
[09:18:17] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1056' (T431682)
[09:18:24] <stashbot>	 T431682: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682
[09:21:14] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-cli
[09:22:06] <wikibugs>	 (03approved) 10dcaro: refactor: rename CommonJob to CommonOptions [repos/cloud/toolforge/jobs-api] (move_cmd_out_of_common_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/367 (https://phabricator.wikimedia.org/T434166) (owner: 10raymond-ndibe)
[09:23:21] <wikibugs>	 (03update) 10dcaro: refactor: rename CommonJob to CommonOptions [repos/cloud/toolforge/jobs-api] (move_cmd_out_of_common_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/367 (https://phabricator.wikimedia.org/T434166) (owner: 10raymond-ndibe)
[09:24:48] <wikibugs>	 (03approved) 10dcaro: d/changelog: bump to 0.3.9 [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/72 (https://phabricator.wikimedia.org/T408783 https://phabricator.wikimedia.org/T434050)
[09:24:59] <wikibugs>	 (03merge) 10dcaro: d/changelog: bump to 0.3.9 [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/72 (https://phabricator.wikimedia.org/T408783 https://phabricator.wikimedia.org/T434050)
[09:26:47] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for all workers
[09:27:50] <wikibugs>	 (03merge) 10dcaro: bump_version: move to trixie [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/118
[09:28:30] <wikibugs>	 (03open) 10dcaro: d/changelog: bump to 0.103.27 [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/119 (https://phabricator.wikimedia.org/T434050)
[09:30:04] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1057' (T431682)
[09:30:11] <stashbot>	 T431682: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682
[09:34:03] <wmcs-alerts>	 FIRING: InstanceDown: Project tools instance tools-k8s-worker-nfs-9 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[09:36:56] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_admin_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[09:39:03] <wmcs-alerts>	 RESOLVED: InstanceDown: Project tools instance tools-k8s-worker-nfs-9 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[09:40:24] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 admin START - Cookbook wmcs.ceph.roll_reboot_osds
[09:40:33] <wmcs-alerts>	 FIRING: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-83 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[09:41:53] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli
[09:41:56] <wmcs-alerts>	 RESOLVED: [2x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_admin_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[09:43:08] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli
[09:45:33] <wmcs-alerts>	 RESOLVED: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-83 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[09:46:23] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1057' (T431682)
[09:46:29] <stashbot>	 T431682: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682
[09:51:33] <wmcs-alerts>	 FIRING: [3x] InstanceDown: Project tools instance tools-k8s-worker-nfs-82 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[09:54:18] <wmcs-alerts>	 RESOLVED: [3x] InstanceDown: Project tools instance tools-k8s-worker-nfs-82 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[09:58:49] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli
[09:59:18] <wmcs-alerts>	 FIRING: [3x] InstanceDown: Project tools instance tools-k8s-worker-nfs-81 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[10:00:08] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli
[10:00:54] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.roll_reboot_osds (exit_code=0)
[10:01:33] <wmcs-alerts>	 RESOLVED: [3x] InstanceDown: Project tools instance tools-k8s-worker-nfs-81 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[10:08:33] <wmcs-alerts>	 FIRING: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-79 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[10:09:18] <wmcs-alerts>	 FIRING: [3x] InstanceDown: Project tools instance tools-k8s-worker-nfs-79 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[10:13:33] <wmcs-alerts>	 RESOLVED: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-79 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[10:19:33] <wmcs-alerts>	 FIRING: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-75 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[10:21:22] <wikibugs>	 (03merge) 10countcount: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/29 (owner: 10renovatebot)
[10:21:33] <wikibugs>	 (03merge) 10countcount: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/46 (owner: 10renovatebot)
[10:21:39] <wikibugs>	 (03merge) 10countcount: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/14 (owner: 10renovatebot)
[10:21:45] <wikibugs>	 (03merge) 10countcount: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/19 (owner: 10renovatebot)
[10:21:48] <wikibugs>	 (03merge) 10countcount: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/6 (owner: 10renovatebot)
[10:21:58] <wikibugs>	 (03merge) 10countcount: Replace hand-rolled PHP deserialization with phpserz [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/76
[10:24:18] <wmcs-alerts>	 RESOLVED: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-75 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[10:35:21] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli
[10:36:59] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli
[10:39:39] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli
[10:40:55] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli
[10:41:07] <wikibugs>	 (03approved) 10dcaro: d/changelog: bump to 0.103.27 [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/119 (https://phabricator.wikimedia.org/T434050)
[10:41:15] <wikibugs>	 (03merge) 10dcaro: d/changelog: bump to 0.103.27 [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/119 (https://phabricator.wikimedia.org/T434050)
[10:41:17] <wikibugs>	 (03approved) 10tlepage: deployment: added certificate with client rights [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/365 (owner: 10dcaro)
[10:42:33] <wmcs-alerts>	 FIRING: [3x] InstanceDown: Project tools instance tools-k8s-worker-nfs-71 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[10:42:44] <wikibugs>	 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [clis] use trixie to bump versions - https://phabricator.wikimedia.org/T434050#12211552 (10dcaro)
[10:43:11] <wikibugs>	 10Toolforge, 06tools-platform-team, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project, 07OKR-Work: [jobs-api] add certificate to authenticate against logs-api - https://phabricator.wikimedia.org/T434120#12211555 (10TLepage-WMF) MR: https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api...
[10:43:36] <wikibugs>	 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [clis] use trixie to bump versions - https://phabricator.wikimedia.org/T434050#12211558 (10dcaro) 05In progress→03Resolved
[10:43:54] <wikibugs>	 10Toolforge, 06tools-platform-team: [all] first of the month dependency upgrades - https://phabricator.wikimedia.org/T433878#12211567 (10dcaro) 05In progress→03Resolved
[10:45:00] <wikibugs>	 (03update) 10dcaro: deployment: added certificate with client rights [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/365
[10:45:15] <wikibugs>	 10Toolforge, 06tools-platform-team, 07OKR-Work: [api-gateway] Specify allowed urls for superusers - https://phabricator.wikimedia.org/T433930#12211571 (10TLepage-WMF) MR: https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/103
[10:46:04] <wikibugs>	 (03update) 10dcaro: deployment: added certificate with client rights [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/365
[10:46:48] <wmcs-alerts>	 RESOLVED: [3x] InstanceDown: Project tools instance tools-k8s-worker-nfs-71 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[10:49:33] <wmcs-alerts>	 FIRING: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-70 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[10:51:48] <wmcs-alerts>	 FIRING: [3x] InstanceDown: Project tools instance tools-k8s-worker-nfs-70 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[10:54:33] <wmcs-alerts>	 RESOLVED: [3x] InstanceDown: Project tools instance tools-k8s-worker-nfs-70 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[10:57:48] <wmcs-alerts>	 FIRING: [3x] InstanceDown: Project tools instance tools-k8s-worker-nfs-69 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[11:00:33] <wmcs-alerts>	 RESOLVED: [3x] InstanceDown: Project tools instance tools-k8s-worker-nfs-69 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[11:05:33] <wmcs-alerts>	 FIRING: [3x] InstanceDown: Project tools instance tools-k8s-worker-nfs-68 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[11:07:33] <wikibugs>	 (03CR) 10Majavah: [C:03+2] labsauth: Log a LabsUser's name and dn when LdapUser fetch fails [labs/striker] - 10https://gerrit.wikimedia.org/r/1324800 (https://phabricator.wikimedia.org/T434684) (owner: 10BryanDavis)
[11:08:48] <wmcs-alerts>	 RESOLVED: [3x] InstanceDown: Project tools instance tools-k8s-worker-nfs-68 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[11:10:27] <wikibugs>	 (03Merged) 10jenkins-bot: labsauth: Log a LabsUser's name and dn when LdapUser fetch fails [labs/striker] - 10https://gerrit.wikimedia.org/r/1324800 (https://phabricator.wikimedia.org/T434684) (owner: 10BryanDavis)
[11:14:10] <wikibugs>	 (03update) 10renovatebot: Update dependency @napi-rs/wasm-runtime to v1.2.3 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/5
[11:14:38] <wikibugs>	 (03update) 10renovatebot: Update pnpm to v11.21.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/9
[11:14:48] <wmcs-alerts>	 FIRING: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-66 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[11:14:58] <wikibugs>	 (03update) 10renovatebot: Update pnpm to v11.21.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/22
[11:15:33] <wmcs-alerts>	 RESOLVED: [3x] InstanceDown: Project tools instance tools-k8s-worker-nfs-66 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[11:16:16] <wikibugs>	 (03update) 10renovatebot: Update pnpm to v11.21.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/48
[11:17:31] <wikibugs>	 (03update) 10renovatebot: Update pnpm to v11.21.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/33
[11:21:33] <wmcs-alerts>	 FIRING: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-65 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[11:24:48] <wmcs-alerts>	 RESOLVED: [3x] InstanceDown: Project tools instance tools-k8s-worker-nfs-65 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[11:30:48] <wmcs-alerts>	 FIRING: [3x] InstanceDown: Project tools instance tools-k8s-worker-nfs-61 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[11:35:48] <wmcs-alerts>	 RESOLVED: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-61 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[11:47:48] <wmcs-alerts>	 FIRING: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-57 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[11:52:33] <wmcs-alerts>	 RESOLVED: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-57 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[12:01:41] <wikibugs>	 06cloud-services-team, 10Toolforge, 06tools-platform-team: [components-api] Deployment token should not be a GET param - https://phabricator.wikimedia.org/T397712#12211851 (10dcaro) +1 for header, will have to change both api-gateway and builds-api.
[12:02:58] <wikibugs>	 (03PS1) 10Blake: api-gateway: remove deprecated service. [labs/private] - 10https://gerrit.wikimedia.org/r/1315832 (https://phabricator.wikimedia.org/T432460)
[12:03:11] <wikibugs>	 (03approved) 10dcaro: Alloy relabeling [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1371 (owner: 10tlepage)
[12:04:09] <wikibugs>	 (03update) 10dcaro: [T401993] Add deployment descriptions to the CLI [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/90 (https://phabricator.wikimedia.org/T401993) (owner: 10mahveotm)
[12:04:18] <wikibugs>	 (03update) 10dcaro: [T398425] Treat invalid YAML as a user error [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/91 (https://phabricator.wikimedia.org/T398425) (owner: 10mahveotm)
[12:04:41] <wikibugs>	 (03CR) 10Clément Goubert: [C:03+1] api-gateway: remove deprecated service. [labs/private] - 10https://gerrit.wikimedia.org/r/1315832 (https://phabricator.wikimedia.org/T432460) (owner: 10Blake)
[12:05:37] <wikibugs>	 (03PS2) 10Blake: api-gateway: remove deprecated service. [labs/private] - 10https://gerrit.wikimedia.org/r/1315832 (https://phabricator.wikimedia.org/T432460)
[12:06:23] <wikibugs>	 (03CR) 10Blake: [C:03+2] api-gateway: remove deprecated service. [labs/private] - 10https://gerrit.wikimedia.org/r/1315832 (https://phabricator.wikimedia.org/T432460) (owner: 10Blake)
[12:06:59] <wikibugs>	 (03CR) 10Blake: [V:03+2 C:03+2] api-gateway: remove deprecated service. [labs/private] - 10https://gerrit.wikimedia.org/r/1315832 (https://phabricator.wikimedia.org/T432460) (owner: 10Blake)
[12:08:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[12:08:41] <wikibugs>	 10Toolforge, 06tools-platform-team, 07OKR-Work: [api-gateway] Specify allowed urls for superusers - https://phabricator.wikimedia.org/T433930#12211868 (10aputhin)
[12:13:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[12:18:41] <wikibugs>	 10Toolforge, 06tools-platform-team, 07OKR-Work: [loki,alloy,logs-api] reconfigure the labels and add metadata - https://phabricator.wikimedia.org/T432969#12211882 (10TLepage-WMF) MR: https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1371
[12:19:33] <wmcs-alerts>	 FIRING: [3x] InstanceDown: Project tools instance tools-k8s-worker-nfs-5 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[12:20:53] <wikibugs>	 10Toolforge, 06tools-platform-team, 07OKR-Work: [loki,alloy,logs-api] reconfigure the labels and add metadata - https://phabricator.wikimedia.org/T432969#12211899 (10dcaro) >>! In T432969#12211882, @TLepage-WMF wrote: > MR: https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/...
[12:23:23] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 admin START - Cookbook wmcs.ceph.roll_reboot_mons
[12:24:33] <wmcs-alerts>	 RESOLVED: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-5 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[12:33:14] <wikibugs>	 (03update) 10mahveotm: [T397863] Show image name in build details [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/147 (https://phabricator.wikimedia.org/T397863)
[12:34:25] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.roll_reboot_mons (exit_code=0)
[12:35:33] <wmcs-alerts>	 FIRING: [3x] InstanceDown: Project tools instance tools-k8s-worker-nfs-46 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[12:35:55] <wikibugs>	 10Toolforge, 06tools-platform-team, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Maintenance: [components-api] do not expose rebuild_storage at the api level - https://phabricator.wikimedia.org/T434780 (10dcaro) 03NEW p:05Triage→03Low
[12:38:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[12:39:48] <wmcs-alerts>	 RESOLVED: [3x] InstanceDown: Project tools instance tools-k8s-worker-nfs-46 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[12:41:33] <wmcs-alerts>	 FIRING: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-45 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[12:43:38] <wmcs-alerts>	 RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[12:44:48] <wmcs-alerts>	 RESOLVED: [3x] InstanceDown: Project tools instance tools-k8s-worker-nfs-45 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[12:50:48] <wmcs-alerts>	 FIRING: [3x] InstanceDown: Project tools instance tools-k8s-worker-nfs-44 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[12:52:06] <wikibugs>	 (03merge) 10countcount: Update pnpm to v11.21.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/33 (owner: 10renovatebot)
[12:52:20] <wikibugs>	 (03merge) 10countcount: Update pnpm to v11.21.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/48 (owner: 10renovatebot)
[12:52:25] <wikibugs>	 (03merge) 10countcount: Update pnpm to v11.21.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/22 (owner: 10renovatebot)
[12:52:30] <wikibugs>	 (03merge) 10countcount: Update pnpm to v11.21.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/9 (owner: 10renovatebot)
[12:52:58] <wikibugs>	 (03update) 10countcount: Remove redundant brace-expansion override [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/32
[12:54:53] <wikibugs>	 10Striker, 06tools-platform-team: 500 internal error when accessing toolsadmin ssh-keys - https://phabricator.wikimedia.org/T434684#12212058 (10Alaexis) Thanks for looking into it. My  Wikimedia developer account username is also Alaexis.  fwiw, today I have no problem accessing the SSH keys page
[12:55:48] <wmcs-alerts>	 RESOLVED: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-44 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[12:56:31] <wikibugs>	 06cloud-services-team, 10Toolforge, 06tools-infrastructure-team: Upgrade toolsbeta to Kubernetes version 1.33 - https://phabricator.wikimedia.org/T433131#12212063 (10Andrew) After some very helpful chat with dcaro and taavi, I have learned and unlearned some things.  - Most importantly, the upgrade steps her...
[12:56:48] <wikibugs>	 (03merge) 10countcount: Remove redundant brace-expansion override [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/32
[12:57:25] <wikibugs>	 06tools-platform-team, 10Toolhub, 13Patch-For-Review: Add banner about Toolhub Evolved - https://phabricator.wikimedia.org/T434556#12212065 (10aputhin) @Schiste fyi, the banner is currently in review and should be added to current Toolhub home soon.
[13:02:07] <wikibugs>	 (03update) 10countcount: Remove stale pnpm dependency exceptions [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/10
[13:02:23] <wikibugs>	 (03update) 10countcount: Remove stale brace-expansion override [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/49
[13:02:27] <wikibugs>	 (03update) 10countcount: Remove stale pnpm dependency exceptions [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/10
[13:02:34] <wikibugs>	 (03update) 10countcount: Remove stale brace-expansion override [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/49
[13:02:54] <wikibugs>	 (03update) 10countcount: Remove stale pnpm dependency exceptions [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/10
[13:05:50] <wikibugs>	 (03update) 10countcount: Remove stale brace-expansion override [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/49
[13:05:51] <wikibugs>	 (03merge) 10countcount: Remove stale pnpm dependency exceptions [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/10
[13:06:12] <wikibugs>	 (03merge) 10countcount: Remove stale brace-expansion override [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/49
[13:09:33] <wmcs-alerts>	 FIRING: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-39 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[13:09:33] <wikibugs>	 (03open) 10countcount: Update Ruff to 0.16.1 and fix new lint errors [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/39
[13:09:40] <wikibugs>	 (03open) 10countcount: Update Ruff to 0.16.1 and fix lint errors [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/27
[13:09:46] <wikibugs>	 (03open) 10countcount: Update Ruff to 0.16.1 and fix lint errors [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/30
[13:13:58] <wikibugs>	 (03close) 10renovatebot: Update dependency @napi-rs/wasm-runtime to v1.2.3 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/5
[13:13:59] <wikibugs>	 (03update) 10renovatebot: Update dependency @napi-rs/wasm-runtime to v1.2.3 - autoclosed [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/5
[13:14:14] <wikibugs>	 (03update) 10countcount: Update Ruff to 0.16.1 and fix lint errors [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/30
[13:14:26] <wikibugs>	 (03update) 10countcount: Update Ruff to 0.16.1 and fix lint errors [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/30
[13:14:33] <wmcs-alerts>	 RESOLVED: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-39 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[13:24:29] <wikibugs>	 10Toolforge, 06tools-platform-team, 07OKR-Work: [loki,alloy,logs-api] reconfigure the labels and add metadata - https://phabricator.wikimedia.org/T432969#12212187 (10aputhin) >>! In T432969#12211899, @dcaro wrote: >>>! In T432969#12211882, @TLepage-WMF wrote: >> MR: https://gitlab.wikimedia.org/repos/cloud/t...
[13:25:30] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/38
[13:25:57] <wikibugs>	 (03update) 10renovatebot: Update python Docker tag to v3.14.7 [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/18
[13:26:05] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/26
[13:26:21] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/29
[13:26:41] <wikibugs>	 (03merge) 10countcount: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/38 (owner: 10renovatebot)
[13:27:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[13:32:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[13:33:48] <wmcs-alerts>	 FIRING: [3x] InstanceDown: Project tools instance tools-k8s-worker-nfs-35 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[13:34:33] <wmcs-alerts>	 RESOLVED: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-35 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[13:36:52] <wikibugs>	 10Quarry, 06tools-platform-team: feat: Add "Copy link" next to "Download data" - https://phabricator.wikimedia.org/T434787 (10Audiodude) 03NEW
[13:42:51] <wikibugs>	 10Quarry, 06tools-platform-team: CI is red because of Black pinning mixup - https://phabricator.wikimedia.org/T434789 (10Audiodude) 03NEW
[13:43:03] <wikibugs>	 10Quarry, 06tools-platform-team: CI is red because of Black pinning mixup - https://phabricator.wikimedia.org/T434789#12212281 (10Audiodude) a:03Audiodude
[13:48:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[13:48:48] <wmcs-alerts>	 FIRING: [3x] InstanceDown: Project tools instance tools-k8s-worker-nfs-33 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[13:49:33] <wmcs-alerts>	 RESOLVED: [3x] InstanceDown: Project tools instance tools-k8s-worker-nfs-33 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[13:53:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[13:55:33] <wmcs-alerts>	 FIRING: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-32 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[13:56:16] <jinxer-wm>	 FIRING: JobUnavailable: Reduced availability for job blackbox_http in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[13:56:41] <jinxer-wm>	 FIRING: PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://grafana.wikimedia.org/d/GWvEXWDZk/prometheus-server?var-datasource=eqiad%20prometheus%2Fcloud - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted
[13:58:48] <wmcs-alerts>	 RESOLVED: [3x] InstanceDown: Project tools instance tools-k8s-worker-nfs-32 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[14:01:16] <jinxer-wm>	 RESOLVED: JobUnavailable: Reduced availability for job blackbox_http in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[14:06:41] <jinxer-wm>	 FIRING: [2x] PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted  - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted
[14:07:39] <wikibugs>	 10Striker, 06tools-platform-team, 10Diffusion, 10Phabricator, 06Release-Engineering-Team (Doing 😎): Striker: Always set all IO to none and hidden for internal URIs when creating Diffusion repositories - https://phabricator.wikimedia.org/T409341#12212396 (10Aklapper) 05Open→03Resolved https://gerr...
[14:08:30] <wikibugs>	 10Toolforge, 06tools-platform-team, 07OKR-Work: [loki,alloy,logs-api] reconfigure the labels and add metadata - https://phabricator.wikimedia.org/T432969#12212400 (10dcaro) >>! In T432969#12212187, @aputhin wrote: >>>! In T432969#12211899, @dcaro wrote: >>>>! In T432969#12211882, @TLepage-WMF wrote: >>> MR:...
[14:09:59] <wikibugs>	 (03update) 10dcaro: [T397863] Show image name in build details [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/147 (https://phabricator.wikimedia.org/T397863) (owner: 10mahveotm)
[14:10:03] <wikibugs>	 (03approved) 10dcaro: [T397863] Show image name in build details [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/147 (https://phabricator.wikimedia.org/T397863) (owner: 10mahveotm)
[14:11:20] <wikibugs>	 06cloud-services-team, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [builds-cli] Show "image_name" in build details - https://phabricator.wikimedia.org/T397863#12212420 (10dcaro)
[14:11:22] <wikibugs>	 06cloud-services-team, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [builds-cli] Show "image_name" in build details - https://phabricator.wikimedia.org/T397863#12212422 (10dcaro) 05Open→03In progress
[14:11:24] <wikibugs>	 06cloud-services-team, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [builds-cli] Show "image_name" in build details - https://phabricator.wikimedia.org/T397863#12212423 (10dcaro) a:03dcaro
[14:11:27] <wikibugs>	 (03merge) 10dcaro: [T397863] Show image name in build details [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/147 (https://phabricator.wikimedia.org/T397863) (owner: 10mahveotm)
[14:15:44] <wikibugs>	 10Cloud-VPS, 06tools-infrastructure-team, 06Data-Platform-SRE: Remove hdfs fuse mounts from clouddumps - https://phabricator.wikimedia.org/T434794 (10taavi) 03NEW
[14:16:41] <jinxer-wm>	 FIRING: [3x] PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted  - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted
[14:16:44] <wikibugs>	 10Data-Services, 06tools-infrastructure-team, 06Data-Platform-SRE: Remove hdfs fuse mounts from clouddumps - https://phabricator.wikimedia.org/T434794#12212452 (10taavi)
[14:21:41] <jinxer-wm>	 FIRING: [3x] PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted  - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted
[14:21:56] <jinxer-wm>	 FIRING: [3x] PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted  - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted
[14:26:41] <jinxer-wm>	 FIRING: [4x] PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted  - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted
[14:26:50] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 06tools-infrastructure-team: Replace scratch-1.cloudinfra-nfs with a trixie VM - https://phabricator.wikimedia.org/T433426#12212527 (10taavi) 05Open→03Resolved
[14:31:41] <jinxer-wm>	 FIRING: [4x] PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted  - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted
[14:34:48] <wmcs-alerts>	 FIRING: [3x] InstanceDown: Project tools instance tools-k8s-worker-nfs-22 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[14:38:33] <wmcs-alerts>	 RESOLVED: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-22 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[14:41:41] <jinxer-wm>	 FIRING: [3x] PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted  - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted
[14:48:25] <wikibugs>	 10Cloud-VPS, 06tools-infrastructure-team: cloudcephosd1043 drives are very very busy - https://phabricator.wikimedia.org/T434334#12212640 (10Andrew) 05Open→03Resolved Yep, this particular issue is fixed with the bdev revert.
[14:51:41] <jinxer-wm>	 RESOLVED: [2x] PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted  - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted
[14:51:48] <wmcs-alerts>	 FIRING: [3x] InstanceDown: Project tools instance tools-k8s-worker-nfs-17 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[14:55:33] <wmcs-alerts>	 RESOLVED: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-17 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[14:58:44] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Add IPv6 PTR record for mail.wikimedia.az - https://phabricator.wikimedia.org/T432761#12212730 (10Andrew) p:05Triage→03Medium
[14:58:48] <wmcs-alerts>	 FIRING: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-16 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[15:01:33] <wmcs-alerts>	 RESOLVED: [3x] InstanceDown: Project tools instance tools-k8s-worker-nfs-16 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[15:02:12] <wikibugs>	 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team, 07OKR-Work: [metricsinfra] allow custom routing for Toolforge alerts - https://phabricator.wikimedia.org/T434801 (10fnegri) 03NEW
[15:06:33] <wmcs-alerts>	 FIRING: [3x] InstanceDown: Project tools instance tools-k8s-worker-nfs-14 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[15:11:33] <wmcs-alerts>	 RESOLVED: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-14 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[15:11:42] <wikibugs>	 10Toolforge, 06tools-platform-team, 07OKR-Work: Define alert conditions in Prometheus (if possible) - https://phabricator.wikimedia.org/T432863#12212795 (10fnegri) > I will look into these two options tomorrow.  I am hopeful that we can avoid the separate prometheus/alertmanager instances, I created {T434801...
[15:15:33] <wmcs-alerts>	 FIRING: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-13 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[15:18:48] <wmcs-alerts>	 RESOLVED: [3x] InstanceDown: Project tools instance tools-k8s-worker-nfs-13 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[15:25:29] <wikibugs>	 10Toolforge, 06tools-platform-team, 07OKR-Work: [loki,alloy,logs-api] reconfigure the labels and add metadata - https://phabricator.wikimedia.org/T432969#12212850 (10aputhin) >>! In T432969#12212400, @dcaro wrote: > It uses that format! https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_r...
[15:33:18] <wmcs-alerts>	 FIRING: InstanceDown: Project tools instance tools-k8s-worker-nfs-1 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[15:33:33] <wmcs-alerts>	 FIRING: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-1 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[15:38:18] <wmcs-alerts>	 RESOLVED: InstanceDown: Project tools instance tools-k8s-worker-nfs-1 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[15:39:29] <wikibugs>	 10Cloud-VPS (Quota-requests), 10Wikidata: Quota increase request for project pm20database - https://phabricator.wikimedia.org/T433839#12212957 (10Andrew) We discussed this briefly in our meeting today. If you're able to support this workload in toolforge that's the best option -- we can raise total ram/cpu quo...
[15:42:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[15:47:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[15:48:05] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all workers
[16:00:25] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
[16:00:58] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
[16:01:35] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
[16:02:15] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
[16:05:49] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
[16:06:52] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
[16:37:31] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
[16:39:14] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
[16:43:25] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
[16:45:15] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
[17:08:20] <wikibugs>	 (03merge) 10countcount: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/29 (owner: 10renovatebot)
[17:16:00] <wikibugs>	 (03update) 10renovatebot: Update python Docker tag to v3.14.7 [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/18
[17:23:49] <wikibugs>	 10Tool-lexeme-forms, 06translatewiki.net, 10LPL Projects (Ongoing maintenance), 10LPL sprints (S2 2026 August): l10n-bot cannot create GitLab merge requests due to gitlab-ssh migration - https://phabricator.wikimedia.org/T432838#12213433 (10LucasWerkmeister) >>! In T432838#12213313, @abi_ wrote: > ` > Unab...
[17:32:02] <wikibugs>	 10Cloud-VPS, 06tools-infrastructure-team: Advice for Hybrid Architecture for AI Tool - https://phabricator.wikimedia.org/T416590#12213465 (10Odinaldo)  Thank you @Andrew!  I will respond in two stages. The first message contains some container statistics to assess the viability of the hosting of the service in...
[17:35:42] <wikibugs>	 10Cloud-VPS, 06tools-infrastructure-team: Advice for Hybrid Architecture for AI Tool - https://phabricator.wikimedia.org/T416590#12213493 (10Odinaldo) Now this second message describes a potential strategy to make sure we are on the same page @Andrew :  # We run the container with the user communication functi...
[17:38:05] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
[17:38:14] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
[17:39:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[17:44:38] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[17:48:04] <wikibugs>	 10Striker, 06tools-platform-team: 500 internal error when accessing toolsadmin ssh-keys - https://phabricator.wikimedia.org/T434684#12213549 (10bd808) >>! In T434684#12208615, @PacmanD wrote: > FWIW I'm seeing 500 on this page about half of the time; I can constantly repro.  @PacmanD can you still reproduce th...
[17:57:39] <wikibugs>	 (03update) 10danyya: Draft: Resolve T434400 "Dumps ingester crashes because of too many prepared statements (2)" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/109
[17:59:38] <wmcs-alerts>	 RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[18:00:47] <wikibugs>	 (03update) 10danyya: Draft: Resolve T434400 "Dumps ingester crashes because of too many prepared statements (2)" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/109
[18:03:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[18:08:38] <wmcs-alerts>	 RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[18:40:36] <wikibugs>	 10Striker, 06tools-platform-team: 500 internal error when accessing toolsadmin ssh-keys - https://phabricator.wikimedia.org/T434684#12213805 (10PacmanD) I'm no longer able to repro after associating the various types of accounts. I think the association with ldap with Wikimedia developer account likely helped....
[18:51:57] <wikibugs>	 (03update) 10danyya: Draft: Resolve T434400 "Dumps ingester crashes because of too many prepared statements (2)" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/109
[19:45:07] <wikibugs>	 (03update) 10danyya: Draft: Resolve T434400 "Dumps ingester crashes because of too many prepared statements (2)" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/109
[19:46:59] <wikibugs>	 (03update) 10anushka10patel: Remember last-selected license across uploads [toolforge-repos/clip2commons] - 10https://gitlab.wikimedia.org/toolforge-repos/clip2commons/-/merge_requests/11 (https://phabricator.wikimedia.org/T434084)
[19:48:52] <wikibugs>	 (03update) 10danyya: Draft: Resolve T434400 "Dumps ingester crashes because of too many prepared statements (2)" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/109
[19:51:57] <wikibugs>	 10Tool-wikinewsie: Wikinewsie cards to be sorted by type based on Wikidta - https://phabricator.wikimedia.org/T434836 (10Pharos) 03NEW
[20:25:20] <wikibugs>	 10Tool-wikinewsie: Wikinewsie cards to be sorted by type based on Wikidta - https://phabricator.wikimedia.org/T434836#12214110 (10Habst) a:03Habst
[21:05:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[21:10:38] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[21:15:38] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[21:25:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[21:31:33] <wikibugs>	 (03update) 10danyya: Draft: Resolve T434400 "Dumps ingester crashes because of too many prepared statements (2)" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/109
[21:57:53] <wikibugs>	 10Tool-wiki-gender-stats: [TF] Ingest data for August 2026 - https://phabricator.wikimedia.org/T434328#12214291 (10Danya) 05Stalled→03In progress
[22:06:49] <wikibugs>	 10Data-Services, 06tools-platform-team, 10VPS-Projects: NFS shared storage for Cloud VPS project `globaleducation` - https://phabricator.wikimedia.org/T434723#12214332 (10Gabinaluz)
[22:09:28] <wikibugs>	 10Tool-inteGraality: Assess whether integraality works/makese sense for Lexemes - https://phabricator.wikimedia.org/T294889#12214344 (10JeanFred) Invoking @Mahir256 for potential use cases :)
[22:20:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[22:25:38] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[22:30:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[22:33:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[22:43:38] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[22:48:38] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[22:53:38] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[22:58:38] <wmcs-alerts>	 RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[23:12:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[23:22:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[23:23:40] <wikibugs>	 10Data-Services, 06tools-platform-team, 10VPS-Projects: NFS shared storage for Cloud VPS project `globaleducation` - https://phabricator.wikimedia.org/T434723#12214579 (10Gabinaluz) I would like to clarify some points because maybe I didn't provide enough information.  Currently, under average operation, the...
[23:30:12] <wikibugs>	 (03update) 10raymond-ndibe: jobs-api: bump to 0.0.558-20260812180740-04ee66e8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1372 (https://phabricator.wikimedia.org/T434166) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[23:30:14] <wikibugs>	 (03approved) 10raymond-ndibe: jobs-api: bump to 0.0.558-20260812180740-04ee66e8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1372 (https://phabricator.wikimedia.org/T434166) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[23:30:20] <wikibugs>	 (03merge) 10raymond-ndibe: jobs-api: bump to 0.0.558-20260812180740-04ee66e8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1372 (https://phabricator.wikimedia.org/T434166) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[23:34:41] <wikibugs>	 10Data-Services, 06tools-platform-team, 10VPS-Projects: NFS shared storage for Cloud VPS project `globaleducation` - https://phabricator.wikimedia.org/T434723#12214593 (10bd808) I think our [[https://wikitech.wikimedia.org/wiki/Help:Object_storage_user_guide|S3-compatible object storage service]] might be a...