[00:01:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:06:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:14:25] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/17 [00:14:26] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [00:14:34] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/29 [00:14:35] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [00:14:43] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/24 [00:14:43] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [00:15:39] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/56 [00:15:40] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [00:16:23] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/41 [00:16:24] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [00:23:29] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12288104 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host cloudvirt1... [00:24:17] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12288105 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host cloudv... [00:39:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:44:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:45:34] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12288114 (10VRiley-WMF) [00:47:15] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12288116 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host cloudv... [01:00:47] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12288152 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host cloudvirt1... [01:14:26] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/17 [01:14:35] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [01:14:51] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/29 [01:14:58] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [01:15:14] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/24 [01:15:21] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [01:16:28] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/56 [01:16:35] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [01:17:46] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/41 [01:18:25] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [01:24:38] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12288169 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host cloudvirt1... [02:14:17] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/17 [02:14:17] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [02:14:26] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/29 [02:14:26] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [02:14:34] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/24 [02:14:35] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [02:15:31] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/56 [02:15:32] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [02:16:14] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/41 [02:16:14] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [03:53:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:58:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:58:26] 10Tool-Pageviews: Add PagePile as a source for Massviews - https://phabricator.wikimedia.org/T437011 (10SunloungerFrog) 03NEW [06:05:32] 10Tool-Pageviews: Add PetScan as a source for Massviews - https://phabricator.wikimedia.org/T437012 (10SunloungerFrog) 03NEW [06:11:01] (03PS1) 10Lokal Profil: add new ua (uk) parameter to config [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1335657 [06:13:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:14:06] (03CR) 10CI reject: [V:04-1] add new ua (uk) parameter to config [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1335657 (owner: 10Lokal Profil) [06:23:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:28:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:30:08] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12288392 (10fgiunchedi) [06:31:11] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12288393 (10fgiunchedi) a:05VRiley-WMF→03fgiunchedi Hosts are reimaged and I have updated netbox with vlan assignmen... [06:31:22] (03PS1) 10Lokal Profil: Temporarily ignore advisory affecting dev-only linter [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1335672 [06:33:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:50:42] (03CR) 10Jean-Frédéric: [C:03+2] Temporarily ignore advisory affecting dev-only linter [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1335672 (owner: 10Lokal Profil) [07:53:33] (03Merged) 10jenkins-bot: Temporarily ignore advisory affecting dev-only linter [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1335672 (owner: 10Lokal Profil) [07:55:32] (03PS2) 10Lokal Profil: add new ua (uk) parameter to config [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1335657 [08:01:07] (03CR) 10Jean-Frédéric: [C:03+2] add new ua (uk) parameter to config [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1335657 (owner: 10Lokal Profil) [08:04:00] (03Merged) 10jenkins-bot: add new ua (uk) parameter to config [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1335657 (owner: 10Lokal Profil) [08:40:17] 10Cloud-VPS (Project-requests), 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Request creation of cephvmm VPS project - https://phabricator.wikimedia.org/T436647#12288636 (10Gehel) [09:35:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:40:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:47:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:52:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:16:46] FIRING: JobUnavailable: Reduced availability for job blackbox_http in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [10:21:46] RESOLVED: JobUnavailable: Reduced availability for job blackbox_http in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [10:47:46] FIRING: JobUnavailable: Reduced availability for job blackbox_http in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [10:52:25] FIRING: [2x] SystemdUnitFailed: hdfs_rsync_mediawiki_content_history.service on clouddumps1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:52:46] RESOLVED: JobUnavailable: Reduced availability for job blackbox_http in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [11:27:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:36:46] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12289200 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host cloudcephosd1055.eqiad.... [12:12:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:14:35] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/17 [12:14:45] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [12:14:47] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [12:15:02] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/29 [12:15:11] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [12:15:13] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [12:15:28] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/24 [12:15:39] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [12:15:41] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [12:16:50] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/56 [12:17:00] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [12:17:01] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [12:18:15] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/41 [12:19:03] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [12:19:04] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [12:30:54] !log tools.cluebot3 Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33872950476 (https://github.com/cluebotng/component-configs/commits/5308018031c449d9ac37a3c10524e02cee50ec8b) [12:30:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot3/SAL [12:32:41] (03open) 10fnegri: alerts-api: Add new component [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1411 (https://phabricator.wikimedia.org/T435971) [12:40:27] (03update) 10fnegri: alerts-api: Add new component [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1411 (https://phabricator.wikimedia.org/T435971) [12:40:32] (03update) 10fnegri: alerts-api: Add new component [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1411 (https://phabricator.wikimedia.org/T435971) [12:44:46] !log fnegri@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component alerts-api [13:02:52] !log fnegri@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component alerts-api [13:10:13] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387#12289672 (10Andrew) I didn't downgrade the ssd firmware but we did upgrade the raid firmware. No change in slow ops behavior. [13:14:26] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/17 [13:14:57] (03update) 10renovatebot: Update vitest monorepo to v5 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/18 [13:14:58] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [13:15:02] (03open) 10renovatebot: Update vitest monorepo to v5 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/18 [13:15:08] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/29 [13:15:22] (03update) 10renovatebot: Update vitest monorepo to v5 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/30 [13:15:22] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [13:15:28] (03open) 10renovatebot: Update vitest monorepo to v5 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/30 [13:15:31] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/24 [13:15:47] (03update) 10renovatebot: Update vitest monorepo to v5 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/25 [13:15:47] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [13:15:55] (03open) 10renovatebot: Update vitest monorepo to v5 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/25 [13:16:47] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/56 [13:17:02] (03update) 10renovatebot: Update vitest monorepo to v5 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/57 [13:17:03] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [13:17:09] (03open) 10renovatebot: Update vitest monorepo to v5 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/57 [13:18:14] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/41 [13:18:30] (03update) 10renovatebot: Update vitest monorepo to v5 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/42 [13:18:31] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [13:18:37] (03open) 10renovatebot: Update vitest monorepo to v5 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/42 [14:17:08] 10Toolforge (Quota-requests), 06tools-infrastructure-team: Request increased quota for wiki-economics Toolforge tool - https://phabricator.wikimedia.org/T436614#12289965 (10Schiste) The storage quota is only transitional during computing. Sorry my initial message may have not been clear, but computed data is q... [14:19:01] (03approved) 10tlepage: alerts-api: Add new component [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1411 (https://phabricator.wikimedia.org/T435971) (owner: 10fnegri) [14:19:05] (03update) 10tlepage: alerts-api: Add new component [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1411 (https://phabricator.wikimedia.org/T435971) (owner: 10fnegri) [14:24:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:34:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:45:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [14:46:13] 10Tool-wikinewsie: User ability to promote non-categorized Wikinewsie stories to Front Page portal - https://phabricator.wikimedia.org/T436982#12290105 (10Pharos) Also, something like Gloria Steinem's death is definitely New York City news, but it's not easily inferred from the categories she's in. I guess there... [14:47:11] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387#12290118 (10Andrew) Following this guide here: https://oneuptime.com/blog/post/2026-03-31-rook-fix-bluestore-slow-op-aler... [14:50:09] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [14:52:40] FIRING: [2x] SystemdUnitFailed: hdfs_rsync_mediawiki_content_history.service on clouddumps1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:58:35] 10Tool-wikinewsie: Wikinewsie cards sometimes have top cut off on mobile - https://phabricator.wikimedia.org/T432935#12290435 (10Pharos) Update: The current behavior on mobile appears to be that the "View article on Wikipedia" blue button usually appears at the top of the article card on loading, with the header... [16:14:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [16:19:02] 10Toolforge, 06tools-platform-team, 07Epic, 07OKR-Work: [hypothesis] ST5.4.1 Extend logging capabilities - https://phabricator.wikimedia.org/T432564#12290521 (10fnegri) //Note: we are experimenting with cross-posting to Phabricator the weekly report that previously was only posted to Asana, where only few... [16:19:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [16:21:23] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component alerts-api [16:24:39] 10VPS-project-Codesearch: Codesearch: "mediawiki/vendor" is missing from Everything preset - https://phabricator.wikimedia.org/T227704#12290532 (10Tgr) I can see how vendor search results would be annoying for must users, but this is extremely confusing. If it says "Everything", it should contain everything, bec... [16:28:19] 10Tool-Pageviews: Add PagePile as a source for Massviews - https://phabricator.wikimedia.org/T437011#12290540 (10MusikAnimal) I recall PagePile was slated to be replaced by [[ https://meta.wikimedia.org/wiki/GULP | GULP ]]. It looks like that never happened? So long as PagePile is still in widespread use, I am... [16:28:58] 10Tool-Pageviews: Hashtag search in Massviews is broken, just says 'An unknown error occurred when querying Hashtag API' - https://phabricator.wikimedia.org/T256057#12290544 (10MusikAnimal) 05Open→03Resolved [16:31:02] 10Tool-Pageviews: Pageviews Analysis not loading when opening a deep link - https://phabricator.wikimedia.org/T362489#12290554 (10MusikAnimal) 05Open→03Resolved a:03MusikAnimal [16:31:16] 10Tool-Pageviews: Outdated referral traffic info in FAQ - https://phabricator.wikimedia.org/T362535#12290558 (10MusikAnimal) 05Open→03Resolved a:03MusikAnimal [16:31:34] 10Tool-Pageviews: pageviews.wmcloud.org shows "Error querying Pageviews API - Not found" for some pages on ko.wp - https://phabricator.wikimedia.org/T316967#12290562 (10MusikAnimal) 05Open→03Resolved a:03MusikAnimal [16:31:53] 10Tool-Pageviews, 07I18n, 07RTL: the order of dropdowns in Topviews must be reversed in RTL languages - https://phabricator.wikimedia.org/T242706#12290576 (10MusikAnimal) 05Open→03Resolved a:03MusikAnimal [16:32:24] 10Tool-Pageviews: [Userviews] Treat 404 API errors as zero pageviews - https://phabricator.wikimedia.org/T191866#12290582 (10MusikAnimal) 05Open→03Resolved a:03MusikAnimal [16:32:41] 10Tool-Pageviews: Userviews not working for some users - https://phabricator.wikimedia.org/T173118#12290585 (10MusikAnimal) 05Open→03Resolved a:03MusikAnimal [16:32:54] 10Tool-Pageviews: Typeahead should be case insensitive - https://phabricator.wikimedia.org/T167488#12290589 (10MusikAnimal) 05Open→03Resolved a:03MusikAnimal [16:33:14] 10Tool-Pageviews: Remove underscores on showed titles - https://phabricator.wikimedia.org/T164412#12290593 (10MusikAnimal) 05Open→03Resolved a:03MusikAnimal [16:33:36] 10Tool-Pageviews: Upgrade Chart.js to use new logarithmic axis - https://phabricator.wikimedia.org/T160765#12290596 (10MusikAnimal) 05Open→03Invalid We now use Apache eCharts [16:34:15] 10Tool-Pageviews: Make options sticky in list-based apps - https://phabricator.wikimedia.org/T166178#12290599 (10MusikAnimal) 05Open→03Resolved a:03MusikAnimal [16:34:27] 10Tool-Pageviews: Time period cannot be selected on pageviews.wmcloud.org on mobile - https://phabricator.wikimedia.org/T367626#12290603 (10MusikAnimal) 05Open→03Resolved a:03MusikAnimal [16:35:05] 10Tool-Pageviews: Surface and/or include links to WikiNav - https://phabricator.wikimedia.org/T341492#12290606 (10MusikAnimal) 05Open→03Resolved a:03MusikAnimal WikiNav data is shown and linked to when viewing stats for a single page. [16:35:54] 10Tool-Pageviews: Updates differs if "Include redirects" is selected or not - https://phabricator.wikimedia.org/T268805#12290612 (10MusikAnimal) 05Open→03Resolved a:03MusikAnimal Should perform better now in the new version [16:36:16] 10Tool-Pageviews: Add option to include redirects in Massviews - https://phabricator.wikimedia.org/T200256#12290616 (10MusikAnimal) 05Open→03Resolved a:03MusikAnimal [16:39:16] 10Tool-Pageviews: Massviews is creating URLs which cannot be used - https://phabricator.wikimedia.org/T357087#12290619 (10MusikAnimal) 05Open→03Resolved a:03MusikAnimal [16:40:10] 10Tool-Pageviews: "Ranked X of the most-viewed pages for ..." states wrong rank - https://phabricator.wikimedia.org/T154986#12290624 (10MusikAnimal) 05Open→03Resolved a:03MusikAnimal [16:40:40] 10Tool-Pageviews: Build custom topviews API - https://phabricator.wikimedia.org/T155018#12290628 (10MusikAnimal) 05Open→03Declined Not needed [16:43:50] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component alerts-api [16:44:44] (03merge) 10fnegri: alerts-api: Add new component [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1411 (https://phabricator.wikimedia.org/T435971) [16:45:16] 10Tool-Pageviews, 07I18n: GRAMMAR doesn't seem to work in https://pageviews.wmcloud.org/siteviews/url_structure/ - https://phabricator.wikimedia.org/T156576#12290648 (10MusikAnimal) 05Open→03Resolved a:03MusikAnimal I'm sure if this is resolved or not, but it doesn't seem to be within my control eith... [16:48:21] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: Add scaffolding for the Toolforge Alerting Service - https://phabricator.wikimedia.org/T435971#12290653 (10fnegri) 05In progress→03Resolved The scaffolding with the simple "fake poller loop" is now live on toolsbeta and tools, and... [17:08:56] 10Toolforge (Quota-requests), 06tools-infrastructure-team: Request increased quota for wiki-economics Toolforge tool - https://phabricator.wikimedia.org/T436614#12290752 (10fnegri) +1, @komla as you're on clinic duty this week, can you take care of this? To recap, the requested changes are: * Namespace memory... [17:18:30] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Upgrade clouddumps hosts to bookworm/trixie - https://phabricator.wikimedia.org/T403154#12290759 (10bking) Just updating the task as I said we would be ready this week. We're still blocked unfort... [17:31:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:35:19] (03update) 10danyya: Draft: Resolve T434400 "Dumps ingester crashes because of too many prepared statements (2)" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/109 [17:36:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:56:02] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387#12290858 (10Andrew) I've applied the same transient setting on 1046 and 1047. [18:00:53] 10Toolforge, 06tools-platform-team, 07OKR-Work: Create template for maintainer alerts emails - https://phabricator.wikimedia.org/T437081 (10fnegri) 03NEW [18:06:33] 06tools-platform-team, 13Patch-For-Review: jobs-api: job with failing healthcheck should not be marked running in the new status implementation - https://phabricator.wikimedia.org/T436726#12290912 (10fnegri) 05Open→03In progress [18:08:41] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-mwlog02 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436469#12290915 (10Southparkfan) 05Open→03In progress [18:22:25] RESOLVED: [2x] SystemdUnitFailed: hdfs_rsync_mediawiki_content_history.service on clouddumps1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:24:05] 10Toolforge, 06tools-platform-team, 07Epic, 07OKR-Work: ST5.4.2 Toolforge Alerting System - https://phabricator.wikimedia.org/T432860#12290935 (10fnegri) //Note: we are experimenting with cross-posting to Phabricator the weekly report that previously was only posted to Asana, where only few people have acc... [18:25:18] 10Tool-Pageviews: Add PetScan as a source for Massviews - https://phabricator.wikimedia.org/T437012#12290943 (10MusikAnimal) It's not clear to met how to programmatically create a JSON dataset from a PetScan result. Using the JSON format option doesn't produce a URL, but I see in the response I can get a [[ http... [18:38:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:41:42] 10Tool-Pageviews: Add PetScan as a source for Massviews - https://phabricator.wikimedia.org/T437012#12290973 (10SunloungerFrog) Whenever one runs a Petscan query, that query and all its options is stored against the PSID. Running the same query by specifying the PSID retrieves all the options. Those options can... [18:53:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:30:39] 10Tool-wsindex, 10Wikisource Reader App (Android): Add descriptions for books where Wikipedia articles are available [API] - https://phabricator.wikimedia.org/T430400#12291099 (10Bodhisattwa) [19:31:14] 10Tool-wsindex, 10Wikisource Reader App (Android): Add summary statistics to analytics page - https://phabricator.wikimedia.org/T430399#12291105 (10Bodhisattwa) [19:33:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:34:28] 10Tool-Pageviews: Add PetScan as a source for Massviews - https://phabricator.wikimedia.org/T437012#12291130 (10MusikAnimal) 05Open→03In progress a:03MusikAnimal Awesome, thank you! I'll get this added. [19:43:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:47:44] 10Cloud-VPS (Project-requests), 06MediaWiki-Media-Platform-Team: Request creation of imagor VPS project - https://phabricator.wikimedia.org/T437088 (10Ladsgroup) 03NEW [20:14:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:49:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:05:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:10:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:24:30] (03CR) 10Krinkle: [C:03+2] Program: Remove send_message_multi in favor of IrcClient.send_message [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334750 (owner: 10Krinkle) [21:25:48] (03Merged) 10jenkins-bot: Program: Remove send_message_multi in favor of IrcClient.send_message [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334750 (owner: 10Krinkle) [21:30:48] 10Tool-wikinewsie: Adjustable image alignment on Wikinewsie - https://phabricator.wikimedia.org/T434467#12291550 (10waldyrious) It seems to me that the main issue is with images that have a transparent background, right? [21:32:10] 10Tool-Pageviews: Add PetScan as a source for Massviews - https://phabricator.wikimedia.org/T437012#12291564 (10MusikAnimal) >>! In T437012#12290973, @SunloungerFrog wrote: > Whenever one runs a Petscan query, that query and all its options is stored against the PSID. Running the same query by specifying the PSI... [21:55:46] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 13Patch-For-Review: Add support for x1 and x4 sections on wiki replicas on the load balancer layer - https://phabricator.wikimedia.org/T409560#12291637 (10bd808) >>! In T423980#12291608, @bd808 wrote: > https://lists.wikimedia.org/hyperkitty/li... [22:08:38] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 13Patch-For-Review: Add support for x1 and x4 sections on wiki replicas on the load balancer layer - https://phabricator.wikimedia.org/T409560#12291654 (10taavi) >>! In T409560#12291637, @bd808 wrote: > Is the x4 stuff all done here? This task... [22:12:43] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-mwlog02 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436469#12291658 (10Southparkfan) Rough outline: - Turn up `deployment-mwlog03` with Trixie (T417002) and a 20 GiB Cinder volume for `/srv` - Adjust `u... [22:14:28] (03PS1) 10Krinkle: IrcClient: Remove unused priorities and improve test coverage [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1336130 [22:14:28] (03PS1) 10Krinkle: Program: Add send queue length to "status" response [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1336131 [22:14:30] (03PS1) 10Krinkle: IrcClient: Increase send_delay from 0.4s to 0.5s [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1336132 [22:17:51] 10Tool-wikinewsie: Auto-Deploy Wikinewsie Commits - https://phabricator.wikimedia.org/T435735#12291663 (10ZhaoFJx) [22:19:18] 10Tool-wikinewsie: Auto-Deploy Wikinewsie Commits - https://phabricator.wikimedia.org/T435735#12291665 (10ZhaoFJx) Credits to @Habst for the work by applying [[ https://wikitech.wikimedia.org/wiki/Help:Toolforge/Deploy_your_tool | Toolforge Components Service ]]! [22:19:22] 10Tool-wikinewsie: Auto-Deploy Wikinewsie Commits - https://phabricator.wikimedia.org/T435735#12291667 (10ZhaoFJx) 05In progress→03Resolved [22:56:22] 10Cloud-VPS (Quota-requests): Quota increase request for project deployment-prep - https://phabricator.wikimedia.org/T437117 (10Southparkfan) 03NEW [22:56:39] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Replace deployment-mwlog02 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436469#12291759 (10Southparkfan) I couldn't finish the Tofu run, because we have too many [[ https://wikitech.wikimedia.org/wiki... [22:56:49] 10Cloud-VPS (Quota-requests): Quota increase request for project deployment-prep - https://phabricator.wikimedia.org/T437117#12291761 (10Southparkfan) [22:57:12] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Replace deployment-mwlog02 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436469#12291762 (10Southparkfan) 05In progress→03Stalled [22:58:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:03:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:04:00] 10Tool-Pageviews: Add PetScan as a source for Massviews - https://phabricator.wikimedia.org/T437012#12291770 (10SunloungerFrog) That looks awesome. You can try 52761149, which is just under 400 pages, and the query runs quickly. On selecting a project, the principal page of Petscan (the Categories tab) requires... [23:35:19] (03CR) 10Krinkle: [C:03+2] IrcClient: Remove unused priorities and improve test coverage [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1336130 (owner: 10Krinkle) [23:35:22] (03CR) 10Krinkle: [C:03+2] Program: Add send queue length to "status" response [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1336131 (owner: 10Krinkle) [23:35:25] (03CR) 10Krinkle: [C:03+2] IrcClient: Increase send_delay from 0.4s to 0.5s [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1336132 (owner: 10Krinkle) [23:36:35] (03Merged) 10jenkins-bot: IrcClient: Remove unused priorities and improve test coverage [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1336130 (owner: 10Krinkle) [23:36:47] (03Merged) 10jenkins-bot: Program: Add send queue length to "status" response [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1336131 (owner: 10Krinkle) [23:36:49] (03Merged) 10jenkins-bot: IrcClient: Increase send_delay from 0.4s to 0.5s [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1336132 (owner: 10Krinkle) [23:52:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:55:50] (03update) 10mahveotm: cancel: split latest deployment endpoint [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/182 (https://phabricator.wikimedia.org/T434499) [23:57:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown