[00:55:07] 10Toolforge, 10cloud-services-team: Do something to Toolforge tools with no non-blocked maintainers - https://phabricator.wikimedia.org/T320342 (10bd808) [02:16:22] 10Toolforge (Toolforge iteration 03), 10Toolforge Build Service, 10Patch-For-Review, 10User-Raymond_Ndibe: [builds-api,logs] Increase pod starting timeout to the same as the request - https://phabricator.wikimedia.org/T354856 (10CodeReviewBot) raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/... [02:17:01] 10Toolforge (Toolforge iteration 03), 10Patch-For-Review: [ci] Add shellcheck to pre-commit where missing - https://phabricator.wikimedia.org/T353052 (10CodeReviewBot) raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/175 [toolforge-deploy] add shellcheck [02:17:23] 10Toolforge (Toolforge iteration 03), 10Patch-For-Review: [ci] Add shellcheck to pre-commit where missing - https://phabricator.wikimedia.org/T353052 (10CodeReviewBot) raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/51 [builds-cli] add shellcheck [02:17:50] 10Toolforge (Toolforge iteration 03), 10Patch-For-Review: [ci] Add shellcheck to pre-commit where missing - https://phabricator.wikimedia.org/T353052 (10CodeReviewBot) raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/22 [envvars-cli] add shellcheck [02:31:26] 10Cloud-Services: Horizon should warn you about not being able to log into a new instance - https://phabricator.wikimedia.org/T354918 (10RoySmith) The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and replace it with... [02:32:28] 10Cloud-VPS: Horizon should warn you about not being able to log into a new instance - https://phabricator.wikimedia.org/T354918 (10RoySmith) [02:57:41] 10Horizon: Horizon should warn you about not being able to log into a new instance - https://phabricator.wikimedia.org/T354918 (10JJMC89) [04:06:56] (SystemdUnitDown) firing: The service unit purge_vm_rbd_images.service is in failed status on host cloudcontrol1005. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1005 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [05:12:22] (HAProxyBackendUnavailable) firing: HAProxy service neutron-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [05:17:22] (HAProxyBackendUnavailable) resolved: HAProxy service neutron-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [06:01:56] (SystemdUnitDown) firing: The systemd unit purge_vm_rbd_images.service on node cloudcontrol1005 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1005 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [06:02:02] 10cloud-services-team: SystemdUnitDown Unit purge_vm_rbd_images.service on node cloudcontrol1005 has been down for long. - https://phabricator.wikimedia.org/T354924 (10phaultfinder) [06:08:00] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [06:37:41] (CloudVPSDesignateLeaks) firing: Detected 3 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:42:41] (CloudVPSDesignateLeaks) firing: (2) Detected 3 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:15:00] 10Toolforge (Toolforge iteration 03), 10Patch-For-Review: [bulids-builder,dotnet] The Procfile buildpack is not being injected in the dotnet group - https://phabricator.wikimedia.org/T354831 (10CodeReviewBot) dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/174... [08:19:14] 10Toolforge (Toolforge iteration 03): Create a kubernetes container with mono and dotnet - https://phabricator.wikimedia.org/T311466 (10dcaro) \o/ >>! In T311466#9455011, @Hawkeye7 wrote: > I find it strange that you are looking for the Program.cs file rather than the *.csproj file, which would seem more logica... [08:48:42] (PrometheusRestarted) firing: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://grafana.wikimedia.org/d/GWvEXWDZk/prometheus-server?var-datasource=eqiad%20prometheus%2Fcloud - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted [08:53:42] (PrometheusRestarted) firing: (2) Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted [09:07:28] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'toolforge-builds-cli' version '0.0.12' [09:07:44] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'toolforge-builds-cli' version '0.0.12' [09:08:19] 10Toolforge (Toolforge iteration 03), 10Toolforge Build Service, 10cloud-services-team: builds-cli loses body text from 503 errors - https://phabricator.wikimedia.org/T354727 (10taavi) 05Open→03Resolved [09:08:28] 10Toolforge Build Service, 10cloud-services-team, 10Patch-For-Review: [harbor,trove] Trove DB filled disk and caused toolforge-build to fail as a result - https://phabricator.wikimedia.org/T354714 (10taavi) [09:13:42] (PrometheusRestarted) firing: (2) Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted [09:18:43] (PrometheusRestarted) resolved: (2) Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted [09:47:42] (CloudVPSDesignateLeaks) firing: (2) Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:51:24] ACKNOWLEDGEMENT - SSH on cloudvirt1063 is CRITICAL: CRITICAL - Socket timeout after 10 seconds Majavah still having CPU issues https://wikitech.wikimedia.org/wiki/SSH/monitoring [09:51:24] ACKNOWLEDGEMENT - Host cloudvirt1063 is DOWN: PING CRITICAL - Packet loss = 100% Majavah still having CPU issues [09:52:42] (CloudVPSDesignateLeaks) resolved: (2) Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:01:58] (SystemdUnitDown) firing: The systemd unit purge_vm_rbd_images.service on node cloudcontrol1005 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1005 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [10:08:16] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [10:17:49] (03PS1) 10Urbanecm: Update election year [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/989861 [10:18:05] (03CR) 10Urbanecm: [C: 03+2] Update election year [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/989861 (owner: 10Urbanecm) [10:18:42] (03Merged) 10jenkins-bot: Update election year [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/989861 (owner: 10Urbanecm) [10:55:23] 10Toolforge (Toolforge iteration 03): [envvars-cli] move pytest from tox to pre-commit - https://phabricator.wikimedia.org/T351476 (10Slst2020) 05Declined→03Resolved [10:55:28] 10Toolforge (Toolforge iteration 03): [envvars-cli] move pytest from tox to pre-commit - https://phabricator.wikimedia.org/T351476 (10Slst2020) 05In progress→03Declined [10:55:35] 10Toolforge (Toolforge iteration 03): [envvars-cli] move pytest from tox to pre-commit - https://phabricator.wikimedia.org/T351476 (10Slst2020) 05Resolved→03In progress [10:58:51] 10Toolforge Build Service: [tbs][dev] decide on which kubernetes bootstrapper to focus on between minikube and kind - https://phabricator.wikimedia.org/T347723 (10Slst2020) I think we're increasingly using lima-kilo, where kind has been working fine. If I'm wrong or you feel a strong preference for minikube, ple... [11:14:07] 10Toolforge (Toolforge iteration 03): [lima-kilo] see how much we can strip off if we only support VM-based setup - https://phabricator.wikimedia.org/T354941 (10dcaro) [11:14:14] 10Toolforge (Toolforge iteration 03): [lima-kilo] see how much we can strip off if we only support VM-based setup - https://phabricator.wikimedia.org/T354941 (10dcaro) 05Open→03In progress [11:17:00] 10Toolforge Build Service: [dev][harbor] reconcile harbor install methods - https://phabricator.wikimedia.org/T354942 (10Slst2020) [11:19:05] 10Toolforge Build Service: [ci] Investigate discrepancy between different CI envs - https://phabricator.wikimedia.org/T353044 (10Slst2020) [11:19:35] 10Toolforge Build Service: [ci][builds-cli][envvars-cli] Investigate discrepancy between different CI envs - https://phabricator.wikimedia.org/T353044 (10Slst2020) [12:21:17] (03CR) 10FNegri: [C: 03+2] SAL logging: invert user and project [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/989219 (https://phabricator.wikimedia.org/T346631) (owner: 10FNegri) [12:25:01] (03Merged) 10jenkins-bot: SAL logging: invert user and project [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/989219 (https://phabricator.wikimedia.org/T346631) (owner: 10FNegri) [12:26:39] 10Cloud-VPS, 10cloud-services-team (FY2023/2024-Q1-Q2), 10Stashbot, 10Patch-For-Review: [wmcs-cookbooks] SAL messages are shown differently when logging via wm-bot - https://phabricator.wikimedia.org/T346631 (10fnegri) 05In progress→03Resolved [12:53:29] (03CR) 10FNegri: [C: 03+1] "Setting "bin-copy-environment" is recommended in the official docs [1] to "increase the security of the started process", but I don't see " [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/988498 (https://phabricator.wikimedia.org/T354320) (owner: 10David Caro) [13:28:45] (ProbeDown) firing: (2) Service tools-k8s-haproxy-3:30000 has failed probes (http_admin_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:33:01] (OpenstackAPIResponse) resolved: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [13:33:03] (PuppetAgentNoResources) firing: No Puppet resources found on instance toolsbeta-bastion-6 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [13:38:45] (ProbeDown) resolved: (2) Service tools-k8s-haproxy-3:30000 has failed probes (http_admin_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:51:54] 10PAWS: Remove paws-123-10 cluster - https://phabricator.wikimedia.org/T354946 (10rook) [13:53:32] vivian-rook opened https://github.com/toolforge/paws/pull/364 [13:53:32] 10PAWS: Remove paws-123-10 cluster - https://phabricator.wikimedia.org/T354946 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/paws/pull/364 [14:01:40] vivian-rook closed https://github.com/toolforge/paws/pull/364 [14:02:12] (SystemdUnitDown) firing: The systemd unit purge_vm_rbd_images.service on node cloudcontrol1005 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1005 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [14:02:58] 10PAWS: Remove paws-123-10 cluster - https://phabricator.wikimedia.org/T354946 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/paws/pull/364 [14:03:00] 10PAWS: Remove paws-123-10 cluster - https://phabricator.wikimedia.org/T354946 (10rook) 05Open→03Resolved [14:03:36] 10PAWS: update helm chart and jupyterhub - https://phabricator.wikimedia.org/T354898 (10rook) a:03rook [14:03:37] 10PAWS: update helm chart and jupyterhub - https://phabricator.wikimedia.org/T354898 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/paws/pull/365 [14:03:39] vivian-rook opened https://github.com/toolforge/paws/pull/365 [14:25:03] (PuppetAgentFailure) firing: Puppet agent failure detected on instance tools-k8s-worker-33 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [15:16:22] (HAProxyBackendUnavailable) firing: HAProxy service neutron-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [15:21:22] (HAProxyBackendUnavailable) resolved: HAProxy service neutron-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [15:25:03] (PuppetAgentFailure) resolved: Puppet agent failure detected on instance tools-k8s-worker-33 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [15:48:01] 10Toolforge (Toolforge iteration 03), 10Patch-For-Review: [lima-kilo] see how much we can strip off if we only support VM-based setup - https://phabricator.wikimedia.org/T354941 (10CodeReviewBot) dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/89 vm support only [15:48:05] 10Toolforge (Toolforge iteration 03), 10Patch-For-Review: [lima-kilo] see how much we can strip off if we only support VM-based setup - https://phabricator.wikimedia.org/T354941 (10CodeReviewBot) dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/89 vm support only [15:54:47] 10Cloud-VPS, 10Toolforge, 10cloud-services-team, 10LDAP: Purge rights or update contact info for Cloud VPS and Toolforge members with invalid email addresses - https://phabricator.wikimedia.org/T218239 (10bd808) >>! In T218239#9451266, @Stashbot wrote: > {nav icon=file, name=Mentioned in SAL (#wikimedia-cl... [16:09:46] (ProbeDown) firing: Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-14:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:13:33] 10Toolforge (Toolforge iteration 03), 10Patch-For-Review: [lima-kilo] see how much we can strip off if we only support VM-based setup - https://phabricator.wikimedia.org/T354941 (10CodeReviewBot) dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/90 Draft: Vm support only [16:14:46] (ProbeDown) resolved: Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-14:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:33:03] (PuppetAgentNoResources) firing: No Puppet resources found on instance toolsbeta-bastion-6 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [16:33:52] 10Toolforge (Toolforge iteration 03), 10Patch-For-Review: [lima-kilo] see how much we can strip off if we only support VM-based setup - https://phabricator.wikimedia.org/T354941 (10CodeReviewBot) dcaro updated https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/90 Vm support only [16:34:29] 10Toolforge (Toolforge iteration 03), 10Patch-For-Review: [lima-kilo] see how much we can strip off if we only support VM-based setup - https://phabricator.wikimedia.org/T354941 (10CodeReviewBot) dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/29 setup_harbor: u... [18:02:12] (SystemdUnitDown) firing: The systemd unit purge_vm_rbd_images.service on node cloudcontrol1005 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1005 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [18:11:33] 10Toolforge (Toolforge iteration 03), 10Toolforge Build Service: [dev][harbor] reconcile harbor install methods - https://phabricator.wikimedia.org/T354942 (10dcaro) a:03dcaro [18:14:06] 10Toolforge (Toolforge iteration 03): [envvars-cli] move pytest from tox to pre-commit - https://phabricator.wikimedia.org/T351476 (10dcaro) 05Declined→03Resolved [18:37:22] (HAProxyBackendUnavailable) firing: HAProxy service neutron-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [18:42:22] (HAProxyBackendUnavailable) resolved: HAProxy service neutron-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [19:33:03] (PuppetAgentNoResources) firing: No Puppet resources found on instance toolsbeta-bastion-6 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:45:45] (ProbeDown) firing: Service tools-k8s-haproxy-3:30000 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-3:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:50:45] (ProbeDown) resolved: Service tools-k8s-haproxy-3:30000 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-3:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:02:12] (SystemdUnitDown) firing: The systemd unit purge_vm_rbd_images.service on node cloudcontrol1005 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1005 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [22:33:03] (PuppetAgentNoResources) firing: No Puppet resources found on instance toolsbeta-bastion-6 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:02:46] (ProbeDown) firing: Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-14:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:07:46] (ProbeDown) resolved: Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-14:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown