[08:31:34] FIRING: ToolforgeWebHighErrorRate: High 5xx rate on Toolforge web services #page - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeWebHighErrorRate - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/infra-k8s-haproxy?var-frontend=k8s-ingress-https&var-backend=k8s-ingress-http&var-cluster=prometheus-tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeWebHighErrorRate [08:36:34] RESOLVED: ToolforgeWebHighErrorRate: High 5xx rate on Toolforge web services #page - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeWebHighErrorRate - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/infra-k8s-haproxy?var-frontend=k8s-ingress-https&var-backend=k8s-ingress-http&var-cluster=prometheus-tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeWebHighErrorRate [08:50:34] FIRING: ToolforgeWebHighErrorRate: High 5xx rate on Toolforge web services #page - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeWebHighErrorRate - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/infra-k8s-haproxy?var-frontend=k8s-ingress-https&var-backend=k8s-ingress-http&var-cluster=prometheus-tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeWebHighErrorRate [08:55:34] RESOLVED: ToolforgeWebHighErrorRate: High 5xx rate on Toolforge web services #page - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeWebHighErrorRate - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/infra-k8s-haproxy?var-frontend=k8s-ingress-https&var-backend=k8s-ingress-http&var-cluster=prometheus-tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeWebHighErrorRate [09:06:30] 06cloud-services-team, 10Toolforge: ToolforgeWebHighErrorRate should not page if a single tool is down - https://phabricator.wikimedia.org/T429738 (10fnegri) 03NEW [09:17:12] 06cloud-services-team, 10Toolforge: ToolforgeWebHighErrorRate should not page if a single tool is down - https://phabricator.wikimedia.org/T429738#12038622 (10fnegri) After a few minutes the webservice is in CrashLoopBackOff again: `lang=shell-session tools.scholia@tools-bastion-15:~$ kubectl get all NAME... [10:05:10] 10Cloud-VPS (Debian Bullseye Deprecation), 10WMIT-Infrastructure: Cloud VPS wlmitvisual01: upgrade Debian bullseye -> trixie - https://phabricator.wikimedia.org/T429723#12038631 (10Ysogo) Thank you for alert. Now checking with WMI board the approach to take. We will be shortly back. [10:45:45] 10Toolforge, 06tools-platform-team: [jobs-cli] emits a warning to re-create valid jobs - https://phabricator.wikimedia.org/T429231#12038635 (10Wbm1058) Just a note that this issue is persisting for jobs which were running before it arose, and are still running. `toolforge jobs restart` does not make the w... [11:39:43] 10Tool-curator: Curator: Select only not-uploaded images of a sequence - https://phabricator.wikimedia.org/T426494#12038640 (10DaxServer) 05Open→03In progress a:03DaxServer [18:08:41] FIRING: CloudVPSDesignateLeaks: Detected 8 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:48:41] RESOLVED: CloudVPSDesignateLeaks: Detected 8 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:58:29] FIRING: ToolforgeToolviewsFailed: Toolviews processing failed - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeToolviewsFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeToolviewsFailed [22:13:29] RESOLVED: ToolforgeToolviewsFailed: Toolviews processing failed - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeToolviewsFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeToolviewsFailed