[00:12:51] FIRING: ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:17:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:27:51] RESOLVED: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:29:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:34:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:39:51] RESOLVED: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:40:51] FIRING: ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:45:06] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:45:51] RESOLVED: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:47:51] FIRING: [3x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:50:06] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:52:51] RESOLVED: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:04:29] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22557173037 (https://github.com/cluebotng/component-configs/commits/a414cf552e0a0c0d2c9e9817f922d56a4c899bf6) [01:04:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [01:27:28] FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance extdist-06 in project extdist - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [01:40:51] FIRING: ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:45:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:50:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:55:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:00:51] RESOLVED: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:32:48] (03open) 10samwilson: Add var/log/ to CI artifacts [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/7 [02:39:34] (03merge) 10samwilson: Add var/log/ to CI artifacts [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/7 [02:43:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:53:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:58:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:03:51] RESOLVED: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:42:51] FIRING: [2x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:47:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:52:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:57:51] RESOLVED: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:44:51] FIRING: [3x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:49:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:54:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:59:51] RESOLVED: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:02:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:03:01] 10tool-fast-ec: fast-ec should use a connection pool - https://phabricator.wikimedia.org/T325511#11661669 (10Legoktm) 05Open→03Resolved a:03Legoktm Fixed in https://gitlab.wikimedia.org/toolforge-repos/fast-ec/-/commit/6d825f760b96f9c6004d976aaf2e660d95561639. [05:06:25] 10tool-fast-ec: fast-ec should cache username -> actor_id, user opt-in status - https://phabricator.wikimedia.org/T325503#11661672 (10Legoktm) [05:07:51] RESOLVED: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:08:51] FIRING: ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:13:06] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:13:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:18:06] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:18:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:23:06] RESOLVED: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:32:51] FIRING: ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:33:06] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:37:51] FIRING: [3x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:42:51] RESOLVED: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:44:51] FIRING: ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:49:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:54:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:59:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:14:51] RESOLVED: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:41:51] FIRING: ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:46:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:01:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:06:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:07:06] RESOLVED: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:02:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:07:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:12:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:17:51] RESOLVED: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:20:51] FIRING: [2x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:25:51] RESOLVED: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:26:51] FIRING: [2x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:31:06] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:31:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:36:06] RESOLVED: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:38:51] FIRING: [2x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:41:06] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:43:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:46:06] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:48:51] RESOLVED: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:11:53] (03update) 10dcaro: images: return the hash if it was passed [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/272 [11:19:22] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS: Increased openstack latency and rabbitmq rolling restarts on certificate update - https://phabricator.wikimedia.org/T418444#11662379 (10fgiunchedi) [11:40:15] 10Tool-campwiz-nxt, 10Google-Summer-of-Code (Google Summer of Code (2026)): GSoC 2026: CampWiz NxT Redesign - https://phabricator.wikimedia.org/T414269#11662446 (10KhushbuWagh) Hi @Nokib_Sarkar and @Tiven2240, ​I am Khushbu Wagh, a Full-stack Web Development student with a strong focus on building responsive a... [11:47:51] FIRING: [2x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:52:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:57:51] RESOLVED: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:58:51] FIRING: ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:03:06] FIRING: [3x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:03:52] RESOLVED: [3x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:05:51] FIRING: [3x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:08:06] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:10:51] RESOLVED: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:20:40] (03update) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/34 [12:37:51] FIRING: [3x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:42:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:46:58] 06cloud-services-team, 10Quarry: Quarry should support connecting to alternative extension databases - https://phabricator.wikimedia.org/T395800#11662669 (10Ladsgroup) [12:47:51] FIRING: [3x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:52:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:53:48] (03update) 10dcaro: images: return the hash if it was passed [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/272 [12:57:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:02:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:06:13] (03update) 10dcaro: images: return the hash if it was passed [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/272 [13:07:51] RESOLVED: [3x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:10:51] FIRING: ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:13:06] RESOLVED: [2x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:27:51] FIRING: ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:32:51] RESOLVED: [2x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:34:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:39:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:43:53] (03approved) 10dcaro: images: return the hash if it was passed [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/272 [13:44:15] (03merge) 10dcaro: images: return the hash if it was passed [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/272 [13:44:26] (03update) 10dcaro: [jobs-api] save business models in a DB [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/114 (owner: 10raymond-ndibe) [13:44:31] (03update) 10dcaro: refactor image parsing and handling [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/273 (owner: 10raymond-ndibe) [13:44:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:47:21] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.465-20260302134436-e2f7baf8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1138 (https://phabricator.wikimedia.org/T417503) [13:47:26] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.465-20260302134436-e2f7baf8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1138 (https://phabricator.wikimedia.org/T417503) [13:49:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:54:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:57:20] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [13:57:22] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Persistence, and 3 others: Set up x1 replication to an-redacteddb1001 - https://phabricator.wikimedia.org/T407485#11662960 (10BTullis) a:03BTullis [13:59:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:04:51] FIRING: [3x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:05:41] (03update) 10raymond-ndibe: logs-api,logs: test start, end params for logs-api and jobs-api log endpoint [repos/cloud/toolforge/toolforge-deploy] (add_logs_api_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1141 (https://phabricator.wikimedia.org/T400917) [14:08:47] (03update) 10raymond-ndibe: support start and end params in logs endpoint [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/267 (https://phabricator.wikimedia.org/T400917) [14:08:47] (03update) 10raymond-ndibe: support start and end params in logs endpoint [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/267 (https://phabricator.wikimedia.org/T400917) [14:09:12] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [14:09:20] (03update) 10raymond-ndibe: logs-api,logs: test start, end params for logs-api and jobs-api log endpoint [repos/cloud/toolforge/toolforge-deploy] (add_logs_api_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1141 (https://phabricator.wikimedia.org/T400917) [14:09:48] (03PS14) 10Andrew Bogott: Add a cookbook to rebuild the rabbitmq cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236841 [14:09:51] RESOLVED: [3x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:10:51] FIRING: ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:12:32] (03update) 10raymond-ndibe: logs-api,logs: test start, end params for logs-api and jobs-api log endpoint [repos/cloud/toolforge/toolforge-deploy] (add_logs_api_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1141 (https://phabricator.wikimedia.org/T400917) [14:12:43] (03update) 10raymond-ndibe: logs-api,logs: test start, end params for logs-api and jobs-api log endpoint [repos/cloud/toolforge/toolforge-deploy] (add_logs_api_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1141 (https://phabricator.wikimedia.org/T400917) [14:15:06] RESOLVED: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:16:51] FIRING: ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:20:02] (03update) 10raymond-ndibe: [jobs-api] save business models in a DB [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/114 [14:20:06] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:21:51] RESOLVED: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:25:06] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:27:21] FIRING: [3x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:30:06] FIRING: [3x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:32:21] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:37:21] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:40:06] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:42:21] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:43:59] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [14:45:07] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:47:21] FIRING: [3x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:48:26] (03update) 10dcaro: jobs-api: add jobs version migration script and docs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/746 (https://phabricator.wikimedia.org/T359649) [14:50:06] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:52:21] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:55:06] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:57:13] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [14:57:21] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:00:06] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:02:21] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:03:54] (03approved) 10dcaro: jobs-api: bump to 0.0.465-20260302134436-e2f7baf8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1138 (https://phabricator.wikimedia.org/T417503) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:03:59] (03merge) 10dcaro: jobs-api: bump to 0.0.465-20260302134436-e2f7baf8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1138 (https://phabricator.wikimedia.org/T417503) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:05:06] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:07:21] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:10:06] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:12:21] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:15:06] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:17:21] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:18:15] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: [components-api] updated image is not restarted - https://phabricator.wikimedia.org/T417503#11663267 (10dcaro) 05In progress→03Resolved This is now deployed and tested in toolsbeta and tools \o/ Plea... [15:20:06] RESOLVED: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:22:21] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:25:06] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:27:21] RESOLVED: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:29:51] FIRING: [3x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:32:06] (03update) 10dcaro: jobs-api: add jobs version migration script and docs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/746 (https://phabricator.wikimedia.org/T359649) [15:32:33] (03update) 10dcaro: jobs-api: add jobs version migration script and docs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/746 (https://phabricator.wikimedia.org/T359649) [15:34:51] RESOLVED: [3x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:36:51] FIRING: [3x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:37:17] (03update) 10dcaro: jobs-api: add jobs version migration script and docs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/746 (https://phabricator.wikimedia.org/T359649) [15:38:45] (03update) 10dcaro: jobs-api: add jobs version migration script and docs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/746 (https://phabricator.wikimedia.org/T359649) [15:41:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:43:11] (03update) 10dcaro: jobs-api: add jobs version migration script and docs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/746 (https://phabricator.wikimedia.org/T359649) [15:46:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:47:06] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:51:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:53:20] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/154 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:53:25] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/154 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:53:39] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/104 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:53:46] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/104 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:54:10] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/96 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:54:14] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/96 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:54:18] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/128 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:54:21] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/128 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:54:26] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/274 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:54:30] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/274 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:54:37] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/31 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:54:41] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/31 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:54:47] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/45 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:54:50] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/45 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:54:56] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/82 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:55:03] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/82 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:55:12] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/87 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:55:23] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/87 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:55:23] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/71 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:55:33] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/71 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:55:45] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/38 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:55:57] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/40 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:55:57] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/38 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:56:04] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/37 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:56:07] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/40 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:56:13] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/84 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:56:19] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/37 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:56:28] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/154 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:56:28] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/84 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:56:36] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/146 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:56:37] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/154 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:56:42] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/73 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:56:43] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/146 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:56:51] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/13 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:56:54] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/73 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:57:00] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/13 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:58:21] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/274 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:59:14] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: builds-builder: bump to 0.0.142-20260302155522-7ff17e10 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1145 [15:59:14] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: builds-builder: bump to 0.0.142-20260302155522-7ff17e10 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1145 [16:01:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:02:35] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: logs-api: bump to 0.0.13-20260302155813-879fbad7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1146 [16:02:38] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: volume-admission: bump to 0.0.81-20260302155509-8fc6e819 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1147 [16:02:42] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: logs-api: bump to 0.0.13-20260302155813-879fbad7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1146 [16:02:49] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: volume-admission: bump to 0.0.81-20260302155509-8fc6e819 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1147 [16:03:05] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: api-gateway: bump to 0.0.88-20260302155547-44264f79 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1148 [16:03:16] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: api-gateway: bump to 0.0.88-20260302155547-44264f79 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1148 [16:03:35] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: ingress-admission: bump to 0.0.79-20260302155630-364cb534 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1149 [16:03:58] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: ingress-admission: bump to 0.0.79-20260302155630-364cb534 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1149 [16:04:04] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: envvars-admission: bump to 0.0.38-20260302155454-a88eccb9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1150 [16:04:05] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: maintain-kubeusers: bump to 0.0.191-20260302155714-fe5f1382 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1151 [16:04:06] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: components-api: bump to 0.0.185-20260302155732-667a7608 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1152 [16:04:25] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: envvars-admission: bump to 0.0.38-20260302155454-a88eccb9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1150 [16:04:28] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: maintain-kubeusers: bump to 0.0.191-20260302155714-fe5f1382 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1151 [16:04:33] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: components-api: bump to 0.0.185-20260302155732-667a7608 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1152 [16:04:38] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.466-20260302155936-7a426183 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1153 [16:04:53] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.466-20260302155936-7a426183 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1153 [16:05:57] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: registry-admission: bump to 0.0.72-20260302155658-28381d32 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1154 [16:05:57] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: registry-admission: bump to 0.0.72-20260302155658-28381d32 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1154 [16:06:28] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-emailer: bump to 0.0.76-20260302160225-1707e838 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1155 [16:06:32] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-emailer: bump to 0.0.76-20260302160225-1707e838 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1155 [16:06:39] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: builds-api: bump to 0.0.209-20260302155339-cd236f5b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1156 [16:06:41] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: builds-api: bump to 0.0.209-20260302155339-cd236f5b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1156 [16:06:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:07:19] (03update) 10raymond-ndibe: runtime::diff_with_running_job: temp conditional to force job version upgrade from v1 -> v2 [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/259 (https://phabricator.wikimedia.org/T359649) [16:07:39] (03CR) 10Andrew Bogott: [C:03+2] Add a cookbook to rebuild the rabbitmq cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236841 (owner: 10Andrew Bogott) [16:08:15] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: envvars-api: bump to 0.0.81-20260302155558-9020eb8b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1157 [16:08:22] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: envvars-api: bump to 0.0.81-20260302155558-9020eb8b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1157 [16:11:03] (03Merged) 10jenkins-bot: Add a cookbook to rebuild the rabbitmq cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236841 (owner: 10Andrew Bogott) [16:11:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:12:06] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:14:06] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component volume-admission [16:16:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:17:00] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission [16:18:35] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component volume-admission [16:18:38] (03update) 10ladsgroup: Add option to create the dns switchover patch automatically [toolforge-repos/switchmaster] - 10https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/merge_requests/13 (https://phabricator.wikimedia.org/T399891) [16:21:34] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission [16:21:51] RESOLVED: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:22:02] (03approved) 10dcaro: volume-admission: bump to 0.0.81-20260302155509-8fc6e819 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1147 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:22:13] (03merge) 10dcaro: volume-admission: bump to 0.0.81-20260302155509-8fc6e819 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1147 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:22:24] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component registry-admission [16:23:09] (03approved) 10marostegui: Add option to create the dns switchover patch automatically [toolforge-repos/switchmaster] - 10https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/merge_requests/13 (https://phabricator.wikimedia.org/T399891) (owner: 10ladsgroup) [16:31:13] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission [16:36:06] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component registry-admission [16:39:47] 06cloud-services-team, 06DC-Ops, 10ops-codfw, 10procurement, 06SRE: cloudgw2004-dev service implementation - https://phabricator.wikimedia.org/T418765 (10Andrew) 03NEW [16:39:54] 06cloud-services-team, 06DC-Ops, 10ops-codfw, 10procurement, 06SRE: cloudgw2004-dev service implementation - https://phabricator.wikimedia.org/T418765#11663972 (10Andrew) p:05Triage→03Medium [16:46:13] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission [16:53:51] FIRING: ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:58:51] RESOLVED: ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:06:16] (03approved) 10dcaro: registry-admission: bump to 0.0.72-20260302155658-28381d32 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1154 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:06:20] (03update) 10dcaro: registry-admission: bump to 0.0.72-20260302155658-28381d32 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1154 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:06:51] (03merge) 10dcaro: registry-admission: bump to 0.0.72-20260302155658-28381d32 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1154 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:07:41] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer [17:22:59] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer [17:22:59] (03update) 10ladsgroup: Add option to create the dns switchover patch automatically [toolforge-repos/switchmaster] - 10https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/merge_requests/13 (https://phabricator.wikimedia.org/T399891) [17:23:16] (03merge) 10ladsgroup: Add option to create the dns switchover patch automatically [toolforge-repos/switchmaster] - 10https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/merge_requests/13 (https://phabricator.wikimedia.org/T399891) [17:33:55] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer [17:34:24] (03update) 10dcaro: jobs-emailer: bump to 0.0.76-20260302160225-1707e838 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1155 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:37:21] (03update) 10raymond-ndibe: runtime::diff_with_running_job: temp conditional to force job version upgrade from v1 -> v2 [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/259 (https://phabricator.wikimedia.org/T359649) [17:37:41] (03update) 10raymond-ndibe: runtime::diff_with_running_job: temp conditional to force job version upgrade from v1 -> v2 [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/259 (https://phabricator.wikimedia.org/T359649) [17:52:52] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer [18:12:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:17:51] RESOLVED: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:19:51] FIRING: [3x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:24:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:29:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:30:06] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:34:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:43:42] 10Tool-recaptime-dev, 06Release-Engineering-Team, 10GitLab (Project Migration): Create new GitLab project group: recaptime-dev - https://phabricator.wikimedia.org/T418698#11664921 (10bd808) Hosting the repos for #Toolforge tools outside of the toolsadmin.wikimedia.org managed https://gitlab.wikimedia.org/too... [18:44:19] 10Tool-recaptime-dev, 06Release-Engineering-Team, 10GitLab (Project Migration): Create new GitLab project group: recaptime-dev - https://phabricator.wikimedia.org/T418698#11664928 (10bd808) > I am not sure if I fit within Wikimedia Communities even through I mostly lurk around English Wikipedia (especially t... [18:44:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:49:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:54:51] FIRING: [3x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:55:49] (03approved) 10dcaro: jobs-emailer: bump to 0.0.76-20260302160225-1707e838 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1155 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [18:55:58] (03merge) 10dcaro: jobs-emailer: bump to 0.0.76-20260302160225-1707e838 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1155 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [18:59:51] RESOLVED: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:00:51] FIRING: [3x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:03:57] (03PS1) 10Majavah: Cleanup deprecated typing aliases [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1247130 [19:03:57] (03PS1) 10Majavah: t1: Improve strong typing [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1247131 [19:03:57] (03PS1) 10Majavah: t1: Use f-strings where possible [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1247132 [19:03:57] (03PS1) 10Majavah: t2: Migrate to linktarget [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1247133 [19:03:58] (03PS1) 10Majavah: t6: Migrate to linktarget [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1247134 [19:05:06] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:06:42] (03CR) 10CI reject: [V:04-1] t1: Use f-strings where possible [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1247132 (owner: 10Majavah) [19:06:51] (03CR) 10CI reject: [V:04-1] Cleanup deprecated typing aliases [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1247130 (owner: 10Majavah) [19:06:55] (03CR) 10CI reject: [V:04-1] t1: Improve strong typing [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1247131 (owner: 10Majavah) [19:06:58] (03CR) 10CI reject: [V:04-1] t6: Migrate to linktarget [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1247134 (owner: 10Majavah) [19:07:02] (03CR) 10CI reject: [V:04-1] t2: Migrate to linktarget [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1247133 (owner: 10Majavah) [19:08:01] (03PS2) 10Majavah: Cleanup deprecated typing aliases [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1247130 [19:08:01] (03PS2) 10Majavah: t1: Improve strong typing [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1247131 [19:08:01] (03PS2) 10Majavah: t1: Use f-strings where possible [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1247132 [19:08:01] (03PS2) 10Majavah: t2: Migrate to linktarget [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1247133 [19:08:02] (03PS2) 10Majavah: t6: Migrate to linktarget [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1247134 [19:08:03] (03PS1) 10Majavah: Format with latest Black [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1247137 [19:08:07] (03PS1) 10Majavah: t2: Cleanup edit summary handling [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1247138 [19:10:06] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:10:16] (03CR) 10CI reject: [V:04-1] t1: Use f-strings where possible [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1247132 (owner: 10Majavah) [19:10:18] (03CR) 10CI reject: [V:04-1] t1: Improve strong typing [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1247131 (owner: 10Majavah) [19:10:25] (03CR) 10CI reject: [V:04-1] t6: Migrate to linktarget [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1247134 (owner: 10Majavah) [19:10:30] (03CR) 10CI reject: [V:04-1] t2: Migrate to linktarget [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1247133 (owner: 10Majavah) [19:10:51] RESOLVED: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:10:53] (03CR) 10CI reject: [V:04-1] t2: Cleanup edit summary handling [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1247138 (owner: 10Majavah) [19:10:55] (03CR) 10Majavah: [C:03+2] Format with latest Black [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1247137 (owner: 10Majavah) [19:11:00] (03CR) 10Majavah: [C:03+2] Cleanup deprecated typing aliases [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1247130 (owner: 10Majavah) [19:13:10] (03PS3) 10Majavah: t1: Improve strong typing [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1247131 [19:13:10] (03PS3) 10Majavah: t1: Use f-strings where possible [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1247132 [19:13:10] (03PS3) 10Majavah: t2: Migrate to linktarget [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1247133 [19:13:10] (03PS3) 10Majavah: t6: Migrate to linktarget [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1247134 [19:13:11] (03PS2) 10Majavah: t2: Cleanup edit summary handling [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1247138 [19:13:56] (03Merged) 10jenkins-bot: Format with latest Black [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1247137 (owner: 10Majavah) [19:13:57] (03Merged) 10jenkins-bot: Cleanup deprecated typing aliases [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1247130 (owner: 10Majavah) [19:14:48] (03CR) 10Majavah: [C:03+2] t1: Improve strong typing [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1247131 (owner: 10Majavah) [19:14:53] (03CR) 10Majavah: [C:03+2] t1: Use f-strings where possible [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1247132 (owner: 10Majavah) [19:14:57] (03CR) 10Majavah: [C:03+2] t2: Migrate to linktarget [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1247133 (owner: 10Majavah) [19:15:01] (03CR) 10Majavah: [C:03+2] t6: Migrate to linktarget [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1247134 (owner: 10Majavah) [19:15:05] (03CR) 10Majavah: [C:03+2] t2: Cleanup edit summary handling [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1247138 (owner: 10Majavah) [19:16:51] FIRING: [3x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:17:27] (03Merged) 10jenkins-bot: t1: Improve strong typing [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1247131 (owner: 10Majavah) [19:17:27] (03Merged) 10jenkins-bot: t1: Use f-strings where possible [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1247132 (owner: 10Majavah) [19:17:28] (03Merged) 10jenkins-bot: t2: Migrate to linktarget [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1247133 (owner: 10Majavah) [19:17:29] (03Merged) 10jenkins-bot: t6: Migrate to linktarget [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1247134 (owner: 10Majavah) [19:17:30] (03Merged) 10jenkins-bot: t2: Cleanup edit summary handling [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1247138 (owner: 10Majavah) [19:21:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:26:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:28:01] 06cloud-services-team, 06collaboration-services, 06SRE Observability, 13Patch-For-Review: Move https://status.wikimedia.org/ away from rackspace - https://phabricator.wikimedia.org/T414098#11665169 (10Dzahn) The release for status.wm.org has been deployed to _staging_ on the k8s cluster within the miscweb... [19:32:06] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:36:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:41:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:46:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:52:06] FIRING: [2x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:52:07] (03update) 10raymond-ndibe: jobs-api: add jobs version migration script and docs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/746 (https://phabricator.wikimedia.org/T359649) (owner: 10dcaro) [19:52:13] (03update) 10raymond-ndibe: jobs-api: add jobs version migration script and docs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/746 (https://phabricator.wikimedia.org/T359649) (owner: 10dcaro) [19:52:54] (03update) 10raymond-ndibe: support start and end params in logs endpoints [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/12 (https://phabricator.wikimedia.org/T400917) [19:56:51] FIRING: [3x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:01:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:06:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:11:51] RESOLVED: [3x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:12:51] FIRING: [2x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:17:06] RESOLVED: [2x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:18:51] FIRING: [2x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:23:52] FIRING: [2x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:28:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:29:06] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:33:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:38:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:43:10] (03update) 10raymond-ndibe: support start and end params in logs endpoints [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/12 (https://phabricator.wikimedia.org/T400917) [20:43:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:48:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:53:12] (03update) 10raymond-ndibe: support since and until params in logs endpoints [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/12 (https://phabricator.wikimedia.org/T400917) [20:53:28] (03update) 10raymond-ndibe: logs-api,logs: test start, end params for logs-api and jobs-api log endpoint [repos/cloud/toolforge/toolforge-deploy] (add_logs_api_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1141 (https://phabricator.wikimedia.org/T400917) [20:53:51] RESOLVED: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:54:21] FIRING: ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:54:55] (03update) 10raymond-ndibe: support since and until params in logs endpoints [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/12 (https://phabricator.wikimedia.org/T400917) [20:56:38] (03CR) 10Eevans: [V:03+2 C:03+2] Add (phony) password for linked_artifacts Cassandra role [labs/private] - 10https://gerrit.wikimedia.org/r/1243986 (https://phabricator.wikimedia.org/T418420) (owner: 10Eevans) [20:59:06] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:59:21] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:04:06] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:05:57] 06cloud-services-team, 10superset.wmcloud.org: Sunset superset.wmcloud.org - https://phabricator.wikimedia.org/T416373#11665536 (10Andrew) > If it is shutting down, it would be good to understand. which will be the available alternatives to provide communities with that kind of support. Quarry accesses the sa... [21:09:06] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:09:21] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:14:06] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:19:06] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:19:21] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:24:06] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:24:21] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:29:06] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:29:21] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:34:06] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:34:21] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:39:06] RESOLVED: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:39:21] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:42:27] (03update) 10raymond-ndibe: support start and end params in logs endpoint [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/267 (https://phabricator.wikimedia.org/T400917) [21:44:06] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:44:34] (03update) 10raymond-ndibe: support start and end params in logs endpoint [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/267 (https://phabricator.wikimedia.org/T400917) [21:49:06] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:49:21] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:54:21] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:59:06] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:59:21] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:04:21] RESOLVED: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:09:06] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:10:21] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:14:06] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:15:21] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:19:06] RESOLVED: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:20:21] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:24:06] RESOLVED: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:27:51] FIRING: [2x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:29:06] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:32:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:37:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:39:06] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:42:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:47:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:49:06] RESOLVED: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:52:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:54:06] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:57:51] RESOLVED: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:59:51] FIRING: ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:04:06] RESOLVED: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:13:23] 10Cloud-VPS (Quota-requests): Quota increases for gitlab-runners - https://phabricator.wikimedia.org/T418813 (10dduvall) 03NEW [23:13:28] 10Cloud-VPS (Quota-requests): Quota increases for gitlab-runners - https://phabricator.wikimedia.org/T418813#11666039 (10dduvall) [23:13:53] 10Cloud-VPS (Quota-requests): Quota increases for gitlab-runners - https://phabricator.wikimedia.org/T418813#11666045 (10dduvall) [23:16:21] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:21:21] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:26:21] RESOLVED: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:32:36] FIRING: [2x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:37:21] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:37:36] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:41:02] (03update) 10raymond-ndibe: [cli] support start and end params in logs [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/145 (https://phabricator.wikimedia.org/T400917) [23:42:22] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:42:36] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:43:16] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/312 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [23:43:19] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/312 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [23:43:25] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/312 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [23:43:33] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/61 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [23:43:39] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/61 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [23:43:42] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/61 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [23:43:51] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/61 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [23:43:53] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/61 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [23:43:56] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/61 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [23:44:12] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/41 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [23:44:19] (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/41 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [23:44:19] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/41 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [23:44:44] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/155 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [23:44:49] (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/155 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [23:44:58] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/155 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [23:45:27] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/74 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [23:45:31] (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/74 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [23:45:40] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/74 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [23:46:08] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/14 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [23:46:12] (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/14 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [23:46:14] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/14 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [23:46:31] (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/41 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [23:46:55] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: maintain-harbor: bump to 0.0.68-20260302234411-b765488a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1158 [23:47:14] (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/74 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [23:47:21] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:47:35] (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/155 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [23:47:36] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:48:16] (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/14 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [23:48:16] (03update) 10raymond-ndibe: [cli] support since and until params in logs [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/145 (https://phabricator.wikimedia.org/T400917) [23:48:36] (03update) 10raymond-ndibe: [cli] support since and until params in logs [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/145 (https://phabricator.wikimedia.org/T400917) [23:48:52] (03update) 10raymond-ndibe: [cli] support since and until params in logs [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/145 (https://phabricator.wikimedia.org/T400917) [23:49:10] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-emailer: bump to 0.0.77-20260302234644-7259b5e1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1159 [23:50:39] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: logs-api: bump to 0.0.14-20260302234841-60bec639 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1146 [23:50:45] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: logs-api: bump to 0.0.14-20260302234841-60bec639 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1146 [23:50:48] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: components-api: bump to 0.0.186-20260302234750-21efe557 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1152 [23:50:55] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: components-api: bump to 0.0.186-20260302234750-21efe557 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1152 [23:52:21] RESOLVED: [3x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown