[00:01:41] 10Tool-wikinewsie: Wikinewsie portals flexibility and user-generated content - https://phabricator.wikimedia.org/T434980#12239776 (10Habst) 05Open→03In progress a:03Habst [00:02:27] 10Tool-wikinewsie: Wikinewsie portals flexibility and user-generated content - https://phabricator.wikimedia.org/T434980#12239795 (10Habst) 05In progress→03Resolved [00:02:29] 10Tool-wikinewsie: Wikinewsie portals flexibility and user-generated content - https://phabricator.wikimedia.org/T434980#12239796 (10Habst) Done, see https://meta.wikimedia.org/wiki/User:Habst/Wikinewsie.json [00:03:18] 10Tool-wikinewsie: Wikinewsie portals flexibility and user-generated content - https://phabricator.wikimedia.org/T434980#12239797 (10Habst) It's re-fetched by server every 5 minutes. [00:04:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:08:51] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387) [00:08:52] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387) [00:08:57] T429387: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387 [00:09:11] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387) [00:09:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:34:25] FIRING: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter@s6.service on clouddb1025:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:46:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:51:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:56:38] RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:00:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:04:44] 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Castor, 10Continuous-Integration-Infrastructure, 06Test Platform (Nairobi 29): Provide more disk I/O headroom for the Castor cache volume (integration-castor06) - https://phabricator.wikimedia.org/T435571 (10Mhurd) 03NEW [01:05:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:06:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:11:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:40:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:45:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:03:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:08:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:13:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:18:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:55:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:00:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:15:19] (03update) 10renovatebot: Update cimg/rust Docker tag to v1.98 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/36 [04:15:23] (03open) 10renovatebot: Update cimg/rust Docker tag to v1.98 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/36 [04:17:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:22:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:34:40] FIRING: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter@s6.service on clouddb1025:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:04:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:09:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:25:42] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387) [06:25:51] T429387: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387 [06:31:57] 10Tool-techcontribs: pie charts with overlapping/unreadable numbers - https://phabricator.wikimedia.org/T384557#12240238 (10Chlod) 05Open→03Resolved a:03Chlod [07:02:46] 10PAWS, 06tools-infrastructure-team, 06tools-platform-team: Make sure dumps-nfs mounts are propagated inside PAWS containers - https://phabricator.wikimedia.org/T432583#12240258 (10fgiunchedi) This is fully deployed, on Monday I'm carrying out a dumps-nfs failover for validation [07:04:50] 10Data-Services, 06tools-infrastructure-team: Clean up clouddumps symlink handling from puppet - https://phabricator.wikimedia.org/T435581 (10fgiunchedi) 03NEW [07:09:14] 10Tool-techcontribs, 10Phabricator-Bot-Requests: Create Phabricator bot for Tech Contribs - https://phabricator.wikimedia.org/T435582 (10Chlod) 03NEW [07:10:00] 10Tool-techcontribs, 10Phabricator-Bot-Requests: Create Phabricator bot for Tech Contribs - https://phabricator.wikimedia.org/T435582#12240293 (10Chlod) [07:15:27] 10Tool-techcontribs, 10Phabricator-Bot-Requests: Create Phabricator bot for Tech Contribs - https://phabricator.wikimedia.org/T435582#12240294 (10Chlod) [07:15:39] 10Tool-techcontribs, 10Phabricator-Bot-Requests: Create Phabricator bot for Tech Contribs - https://phabricator.wikimedia.org/T435582#12240295 (10Chlod) [08:05:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:10:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:18:30] 10Tool-techcontribs, 10Phabricator-Bot-Requests: Create Phabricator bot for Tech Contribs - https://phabricator.wikimedia.org/T435582#12240405 (10taavi) a:03taavi [08:21:13] (03PS11) 10Krinkle: [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) [08:21:42] 10Tool-techcontribs, 10Phabricator-Bot-Requests: Create Phabricator bot for Tech Contribs - https://phabricator.wikimedia.org/T435582#12240410 (10taavi) 05Open→03Resolved Created @TechContribsBot. See P96218 for the API token. [08:21:47] (03CR) 10CI reject: [V:04-1] [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [08:34:40] FIRING: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter@s6.service on clouddb1025:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:50:02] 10Tool-techcontribs, 10Phabricator-Bot-Requests: Create Phabricator bot for Tech Contribs - https://phabricator.wikimedia.org/T435582#12240470 (10Chlod) Thank you! :D [09:03:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:08:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:13:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:32:41] 06cloud-services-team, 10Data-Services, 06tools-platform-team: [wikireplicas] Create views for new wiki minwikiquote - https://phabricator.wikimedia.org/T429946#12240553 (10fnegri) 05Open→03In progress a:03fnegri [09:34:19] 10Data-Services, 06tools-platform-team: [wikireplicas] Create views for new wiki bolwiki - https://phabricator.wikimedia.org/T429954#12240559 (10fnegri) 05Open→03In progress a:03fnegri [09:39:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:44:25] RESOLVED: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter@s6.service on clouddb1025:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:45:15] 10PAWS, 06tools-platform-team: [bug] server won't launch - https://phabricator.wikimedia.org/T435587 (10Jklamo) 03NEW [09:49:11] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06DBA: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12240619 (10fnegri) [09:49:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:54:17] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06DBA: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12240639 (10fnegri) > `lang=shell-session > root@clouddb1025:~# systemctl reset-failed > root@clouddb1025:~# systemctl daemon-reload > ro... [10:16:45] 10Tool-techcontribs: pie charts with overlapping/unreadable numbers - https://phabricator.wikimedia.org/T384557#12240736 (10Bugreporter2) The example "pie chart" in https://www.mediawiki.org/wiki/Extension:Chart isn't very pie-like: {F99377001} What did you do to it? [10:38:11] 10PAWS, 06tools-platform-team: [bug] server won't launch - https://phabricator.wikimedia.org/T435587#12240769 (10tomasz) I'm also affecte here, seems to be the exact same node as @Jklamo: ` 2026-08-21T10:33:59.725843Z [Normal] Successfully assigned prod/jupyter--4fdder-42ot to paws-127d-mq2566hee6zs-node-4 20... [10:40:28] 10Data-Services, 06tools-platform-team: [wikireplicas] Create views for new wiki bolwiki - https://phabricator.wikimedia.org/T429954#12240806 (10fnegri) 05In progress→03Resolved [10:40:29] 06cloud-services-team, 10Data-Services, 06tools-platform-team: [wikireplicas] Create views for new wiki minwikiquote - https://phabricator.wikimedia.org/T429946#12240802 (10fnegri) 05In progress→03Resolved @xcollazo done, that should fix the Sqoop failures. Let me know if you're still having issues. [11:35:12] 10PAWS, 06tools-platform-team: [bug] server won't launch - https://phabricator.wikimedia.org/T435587#12240995 (10fgiunchedi) I'm going to take a look, we did some paws and nfs work yesterday [11:52:05] 10PAWS, 06tools-platform-team: [bug] server won't launch - https://phabricator.wikimedia.org/T435587#12241035 (10fgiunchedi) @Jklamo @tomasz would you mind trying again? node-4 was in trouble with high load, I rebooted the node [11:56:10] 10PAWS, 06tools-platform-team: [bug] server won't launch - https://phabricator.wikimedia.org/T435587#12241059 (10Jklamo) Wokrs OK now. [11:57:54] 10PAWS, 06tools-platform-team: [bug] server won't launch - https://phabricator.wikimedia.org/T435587#12241069 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi Ok thank you, I can spawn my notebook on node-4 too. Resolving [12:06:14] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12241104 (10fgiunchedi) a:05VRiley-WMF→03fgiunchedi >>! In T431682#12239559, @VRiley-WMF wrote: > Okay, so... > > A... [12:16:41] 10Tool-wikimonitor: Implement a real-time visual connection status indicator—a "traffic light" system - https://phabricator.wikimedia.org/T425925#12241169 (10Gerges) 05Open→03Resolved Thanks @Praffq :) [12:32:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:37:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:00:17] 10Tool-wikimonitor: [WikiStreamService] Fix silent SSE stream freeze on half-open TCP drops - https://phabricator.wikimedia.org/T435612 (10Gerges) 03NEW [13:00:43] 10Tool-wikimonitor, 07good first task: [WikiStreamService] Fix silent SSE stream freeze on half-open TCP drops - https://phabricator.wikimedia.org/T435612#12241274 (10Gerges) Thank you for tagging this task with #good_first_task for Wikimedia newcomers! Newcomers often may not be aware of things that may seem... [13:22:48] 10Tool-wiki-gender-stats: Add alternative texts to images on the website - https://phabricator.wikimedia.org/T434311#12241307 (10ShivangiU2) Thanks Danya, I'll definitely ping you for help with French and Esperanto when I get there! [13:24:29] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387#12241309 (10Andrew) I'm now running fio tests on cloudcephosd1041 and 1042. Most stats are pretty similar (1042 is 10-15%... [13:35:22] 06cloud-services-team, 10Tool-paulina, 10Toolforge, 06tools-platform-team: Automated traffic affects paulina.toolforge.org performance - https://phabricator.wikimedia.org/T432878#12241342 (10Pepe_piton) 05In progress→03Resolved [13:35:48] PROBLEM - Host cloudcephosd1042 is DOWN: PING CRITICAL - Packet loss = 100% [13:39:47] FIRING: NodeDown: Node cloudcephosd1042 is down. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcephosd1042 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [13:43:56] RECOVERY - Host cloudcephosd1042 is UP: PING OK - Packet loss = 0%, RTA = 0.24 ms [13:44:47] RESOLVED: NodeDown: Node cloudcephosd1042 is down. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcephosd1042 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [13:48:27] (03open) 10dcaro: Silly commit [toolforge-repos/wm-lol] - 10https://gitlab.wikimedia.org/toolforge-repos/wm-lol/-/merge_requests/1 [13:49:25] FIRING: [8x] SystemdUnitFailed: ceph-osd@161.service on cloudcephosd1042:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:59:54] 10Tool-paulina: Link to create new item if no results - https://phabricator.wikimedia.org/T370223#12241413 (10Pepe_piton) 05Open→03Resolved a:03Pepe_piton @Ademola I merged your contribution. Thank you! [14:02:56] 10Tool-paulina: Improve the Paulina tool through different tasks - https://phabricator.wikimedia.org/T393096#12241425 (10Pepe_piton) 05In progress→03Resolved [14:11:14] PROBLEM - Host cloudcephosd1042 is DOWN: PING CRITICAL - Packet loss = 100% [14:13:56] RECOVERY - Host cloudcephosd1042 is UP: PING OK - Packet loss = 0%, RTA = 0.30 ms [14:14:25] RESOLVED: [8x] SystemdUnitFailed: ceph-osd@161.service on cloudcephosd1042:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:14:35] (03update) 10dcaro: Silly commit [toolforge-repos/wm-lol] - 10https://gitlab.wikimedia.org/toolforge-repos/wm-lol/-/merge_requests/1 [14:14:38] (03update) 10dcaro: Silly commit [toolforge-repos/wm-lol] - 10https://gitlab.wikimedia.org/toolforge-repos/wm-lol/-/merge_requests/1 [14:16:26] PROBLEM - Host cloudcephosd1042 is DOWN: PING CRITICAL - Packet loss = 100% [14:16:47] FIRING: NodeDown: Node cloudcephosd1042 is down. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcephosd1042 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [14:19:45] 06cloud-services-team, 10Data-Services, 06tools-platform-team: [wikireplicas] Create views for new wiki minwikiquote - https://phabricator.wikimedia.org/T429946#12241448 (10xcollazo) Thanks @fnegri! [14:20:56] RECOVERY - Host cloudcephosd1042 is UP: PING OK - Packet loss = 0%, RTA = 0.22 ms [14:21:47] RESOLVED: NodeDown: Node cloudcephosd1042 is down. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcephosd1042 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [14:25:43] 10Tool-paulina: Restore scholary works to author works query results - https://phabricator.wikimedia.org/T397963#12241455 (10Pepe_piton) 05Open→03Resolved a:03Pepe_piton @marfossatti Done. Basically, the [[ https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/WDQS_graph_split | Wikidata graph sp... [14:28:57] FIRING: [8x] SystemdUnitFailed: ceph-osd@161.service on cloudcephosd1042:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:29:44] 10Tool-paulina: Project Proposal: 3-Month Outreachy Project Plan — Add Wikidata Editing Features to Paulina — by Afanyu Lionel - https://phabricator.wikimedia.org/T408889#12241463 (10Pepe_piton) 05Open→03Declined @Afanyulionel I'm closing this task as it was not finally implemented. Thanks anyway! [14:35:24] (03open) 10sebastian-berlin-wmse: Remove commit as an option in help message for `toolforge build start --ref` [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/149 [14:41:46] PROBLEM - Host cloudcephosd1042 is DOWN: PING CRITICAL - Packet loss = 100% [14:46:47] FIRING: NodeDown: Node cloudcephosd1042 is down. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcephosd1042 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [14:53:52] RESOLVED: [8x] SystemdUnitFailed: ceph-osd@161.service on cloudcephosd1042:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:24:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:29:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:34:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:42:57] 10Tool-sentinel: Refresh username/email on OAuth re-login - https://phabricator.wikimedia.org/T431903#12241887 (10AROHI_SINGH) i don't have access of gitlab so am not able to do that this task [15:49:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:59:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:02:10] 06tools-infrastructure-team, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to wmcs-roots for bliviero - https://phabricator.wikimedia.org/T435123#12241959 (10mark) >>! In T435123#12223551, @Andrew wrote: > @mark in theory Belinda is the approver of record for this group but we would l... [16:03:00] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [16:03:04] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [16:04:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:05:42] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy [16:05:43] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) [16:08:36] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [16:08:39] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [16:09:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:10:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [16:11:14] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [16:11:49] 10Toolforge, 06tools-platform-team, 07Epic, 07OKR-Work: Add a dedicated Toolforge alertmanager instance - https://phabricator.wikimedia.org/T435636 (10fnegri) 03NEW [16:14:24] PROBLEM - Host cloudcephosd1042 is DOWN: PING CRITICAL - Packet loss = 100% [16:14:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:16:37] 10Toolforge, 06tools-platform-team, 07OKR-Work: Email tool maintainers when an alert condition is detected - https://phabricator.wikimedia.org/T432865#12242040 (10fnegri) @dcaro I can see two separate concerns in your comment, one is storing configuration in the Metricsinfra db, the other is discovering late... [16:16:48] RECOVERY - Host cloudcephosd1042 is UP: PING OK - Packet loss = 0%, RTA = 0.44 ms [16:18:21] 10Toolforge, 06tools-platform-team, 07OKR-Work: Define alert conditions in Prometheus (if possible) - https://phabricator.wikimedia.org/T432863#12242057 (10fnegri) 05In progress→03Resolved > I'll create a separate task for setting up a dedicated Toolforge alertmanager Task created: {T435636} Everyt... [16:19:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:24:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:29:38] RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:30:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:30:39] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [16:35:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:40:38] RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:58:01] 10Toolforge, 06tools-platform-team, 07Epic, 07OKR-Work: Add a dedicated Toolforge alertmanager instance - https://phabricator.wikimedia.org/T435636#12242169 (10fnegri) a:05fnegri→03None [17:28:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:37:24] 10Tool-wikinewsie: Recent sources for Wikinewsie - https://phabricator.wikimedia.org/T433528#12242288 (10Pharos) This should probably interface with or borrow from Internet Archive's [[ https://citations.scatteranalytics.com/explorer/ | Citation Explorer ]] It will likely also interact with the new AI feature f... [17:38:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:47:51] 10Tool-wikinewsie: GDELT feed for Wikinewsie - https://phabricator.wikimedia.org/T435650 (10Pharos) 03NEW [17:51:52] 10Tool-wiki-gender-stats: Add a humans.txt file - https://phabricator.wikimedia.org/T435652 (10Danya) 03NEW [17:52:21] 10Tool-wiki-gender-stats: Add a humans.txt file - https://phabricator.wikimedia.org/T435652#12242407 (10Danya) [17:52:29] 10Tool-wiki-gender-stats: Add a humans.txt file - https://phabricator.wikimedia.org/T435652#12242409 (10Danya) p:05Triage→03Lowest [18:07:24] 10Tool-techcontribs: pie charts with overlapping/unreadable numbers - https://phabricator.wikimedia.org/T384557#12242440 (10Chlod) @Bugreporter2 I did nothing. Not sure how this is my problem? This is a task for a Toolforge tool; it'd be great if you checked the task's projects beforehand. [18:13:19] 10Tool-techcontribs: pie charts with overlapping/unreadable numbers - https://phabricator.wikimedia.org/T384557#12242456 (10Chlod) Offtopic, but I've fixed the chart: https://commons.wikimedia.org/w/index.php?title=Data:Example.Pie.chart&diff=prev&oldid=1264919181 [18:37:33] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Upgrade clouddumps hosts to bookworm/trixie - https://phabricator.wikimedia.org/T403154#12242486 (10BTullis) a:05BTullis→03bking I believe that @bking will be able to run the upgrades next week. [19:01:10] 06tools-infrastructure-team, 06SRE, 10SRE-Access-Requests: Requesting access to wmcs-roots for bliviero - https://phabricator.wikimedia.org/T435123#12242511 (10Andrew) 05Stalled→03Resolved a:03Andrew she's in! [19:06:49] 10Tool-wikimedia-attribution, 06App Growth, 06MediaWiki-API-Platform-Team, 06WikipediaApp, 07OKR-Work: Make mobile app store badges more visible on Wikipedia.org portal page - https://phabricator.wikimedia.org/T432758#12242520 (10pmiazga) 05Stalled→03Declined In favour of {T429834} [19:52:03] 10Cloud-VPS, 06tools-infrastructure-team: Improve VM backup performance - https://phabricator.wikimedia.org/T434014#12242657 (10Andrew) Adjusting # of parallel reads and writes did not affect job time. [20:09:40] 10Tool-wikinewsie: Extended time window AI edit summaries for Wikinewsie - https://phabricator.wikimedia.org/T435659 (10Pharos) 03NEW [20:12:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:22:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:27:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:37:38] RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:39:27] 10Tool-wheres-my-code-running, 06Release-Engineering-Team (Doing 😎): Where is my change running tool - https://phabricator.wikimedia.org/T434726#12242739 (10bd808) I made a #tool-wheres-my-code-running project with #striker and a stub developer docs page at https://wikitech.wikimedia.org/wiki/Tool:Wheres-m... [20:40:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:41:01] 10Tool-wheres-my-code-running: wheres-my-code-running.git missing license information - https://phabricator.wikimedia.org/T435661 (10bd808) 03NEW [20:45:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:50:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:55:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:57:46] (03open) 10bd808: COPYING: add GPL-2.0-or-later license [toolforge-repos/wheres-my-code-running] - 10https://gitlab.wikimedia.org/toolforge-repos/wheres-my-code-running/-/merge_requests/1 (https://phabricator.wikimedia.org/T435661) [20:57:51] (03update) 10bd808: COPYING: add GPL-2.0-or-later license [toolforge-repos/wheres-my-code-running] - 10https://gitlab.wikimedia.org/toolforge-repos/wheres-my-code-running/-/merge_requests/1 (https://phabricator.wikimedia.org/T435661) [20:59:38] (03approved) 10dancy: COPYING: add GPL-2.0-or-later license [toolforge-repos/wheres-my-code-running] - 10https://gitlab.wikimedia.org/toolforge-repos/wheres-my-code-running/-/merge_requests/1 (https://phabricator.wikimedia.org/T435661) (owner: 10bd808) [21:00:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:00:54] 10Tool-wheres-my-code-running, 13Patch-For-Review, 06Release-Engineering-Team (Doing 😎): wheres-my-code-running.git missing license information - https://phabricator.wikimedia.org/T435661#12242786 (10bd808) 05Open→03In progress a:03bd808 [21:05:38] RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:08:25] 10Tool-wheres-my-code-running: Convert deb install spec from Aptfile to project.toml - https://phabricator.wikimedia.org/T435662 (10bd808) 03NEW [21:27:03] (03open) 10bd808: dev: Convert deb install spec from Aptfile to project.toml [toolforge-repos/wheres-my-code-running] - 10https://gitlab.wikimedia.org/toolforge-repos/wheres-my-code-running/-/merge_requests/2 (https://phabricator.wikimedia.org/T435662) [21:27:10] (03update) 10bd808: dev: Convert deb install spec from Aptfile to project.toml [toolforge-repos/wheres-my-code-running] - 10https://gitlab.wikimedia.org/toolforge-repos/wheres-my-code-running/-/merge_requests/2 (https://phabricator.wikimedia.org/T435662) [21:31:57] 10Tool-wheres-my-code-running, 13Patch-For-Review, 06Release-Engineering-Team (Doing 😎): Convert deb install spec from Aptfile to project.toml - https://phabricator.wikimedia.org/T435662#12242900 (10bd808) 05Open→03In progress [21:36:09] (03approved) 10dancy: dev: Convert deb install spec from Aptfile to project.toml [toolforge-repos/wheres-my-code-running] - 10https://gitlab.wikimedia.org/toolforge-repos/wheres-my-code-running/-/merge_requests/2 (https://phabricator.wikimedia.org/T435662) (owner: 10bd808) [21:46:08] 10Tool-wheres-my-code-running: Setup local and CI linters and tests - https://phabricator.wikimedia.org/T435665 (10bd808) 03NEW [21:57:02] 06Toolforge-standards-committee: Adoption request please. 🪐 - https://phabricator.wikimedia.org/T435666 (10Carrotlandz) 03NEW [22:38:46] (03merge) 10bd808: COPYING: add GPL-2.0-or-later license [toolforge-repos/wheres-my-code-running] - 10https://gitlab.wikimedia.org/toolforge-repos/wheres-my-code-running/-/merge_requests/1 (https://phabricator.wikimedia.org/T435661) [23:22:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:27:38] RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown