[00:36:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:41:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:45:52] 10Cloud-VPS (Project-requests): Request creation of SignWriting VPS project - https://phabricator.wikimedia.org/T441738#12452279 (10Bugreporter) This exists but is not maintained: {T435781} [00:48:47] FIRING: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [00:53:47] RESOLVED: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [00:59:16] FIRING: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [01:09:30] RESOLVED: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [01:27:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:32:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:33:47] FIRING: SystemdUnitFailed: check-private-data.service on clouddb1023:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:35:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:45:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:29:56] 10Tool-autoscout: NPP and admin rebuilds overwrite each other's AfD participation; shared afd_afc_cursor skips the second cohort - https://phabricator.wikimedia.org/T440928#12452469 (10Dreamyshade) I gave this to the LLM, and it attempted to fix the issue: https://gitlab.wikimedia.org/dreamyshade/autoscout/-/com... [04:32:01] 10Tool-autoscout: replace_snapshot deletes every snapshot_meta key, blanking /npp and /admin and resetting the AfD cursor - https://phabricator.wikimedia.org/T440929#12452476 (10Dreamyshade) Worked on this with the other related issues here: https://gitlab.wikimedia.org/dreamyshade/autoscout/-/commit/b1b6b9c9215... [04:32:03] 10Tool-autoscout: Shared DYK/FA bot rows saved by --daily never expire; the weekly refresh reuses rows up to a week old - https://phabricator.wikimedia.org/T440930#12452478 (10Dreamyshade) Worked on this with the other related issues here: https://gitlab.wikimedia.org/dreamyshade/autoscout/-/commit/b1b6b9c9215f5... [04:32:05] 10Tool-autoscout: One failed AI-edit API request aborts the whole full refresh; maxlag gives up after 7 s - https://phabricator.wikimedia.org/T440931#12452480 (10Dreamyshade) Worked on this with the other related issues here: https://gitlab.wikimedia.org/dreamyshade/autoscout/-/commit/b1b6b9c9215f52ee2c2a981cae0... [04:32:09] 10Tool-autoscout: --daily wipes an interrupted weekly run's checkpoint and leaves stale stages that a later daily can reuse - https://phabricator.wikimedia.org/T440932#12452482 (10Dreamyshade) Worked on this with the other related issues here: https://gitlab.wikimedia.org/dreamyshade/autoscout/-/commit/b1b6b9c92... [04:32:31] 10Tool-autoscout: Weekly full refresh likely exceeds its 3600 s job timeout - https://phabricator.wikimedia.org/T440933#12452484 (10Dreamyshade) Worked on this with the other related issues here: https://gitlab.wikimedia.org/dreamyshade/autoscout/-/commit/b1b6b9c9215f52ee2c2a981cae00dc42dcc20881 [04:34:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:36:36] 10Tool-autoscout: NPP and admin rebuilds overwrite each other's AfD participation; shared afd_afc_cursor skips the second cohort - https://phabricator.wikimedia.org/T440928#12452486 (10Audiodude) 05Open→03Resolved a:03Audiodude [04:39:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:40:49] 10Tool-autoscout: Shared DYK/FA bot rows saved by --daily never expire; the weekly refresh reuses rows up to a week old - https://phabricator.wikimedia.org/T440930#12452488 (10Audiodude) 05Open→03Resolved a:03Audiodude [04:45:09] 10Tool-autoscout: One failed AI-edit API request aborts the whole full refresh; maxlag gives up after 7 s - https://phabricator.wikimedia.org/T440931#12452491 (10Audiodude) 05Open→03Resolved a:03Audiodude [04:46:37] 10Tool-autoscout: Weekly full refresh likely exceeds its 3600 s job timeout - https://phabricator.wikimedia.org/T440933#12452493 (10Audiodude) 05Open→03Resolved a:03Audiodude [04:48:06] 10Tool-autoscout: --daily wipes an interrupted weekly run's checkpoint and leaves stale stages that a later daily can reuse - https://phabricator.wikimedia.org/T440932#12452495 (10Audiodude) 05Open→03Resolved a:03Audiodude [05:02:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:03:47] RESOLVED: SystemdUnitFailed: check-private-data.service on clouddb1023:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:07:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:25:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:30:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:18:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:18:59] 10Cloud-VPS (Project-requests): Request creation of SignWriting VPS project - https://phabricator.wikimedia.org/T441738#12452606 (10Aklapper) > To host a basic Wikimedia mediaserver 1.46.2 Such a product does not exist. > and an extension I developed Who would use that new instance and for what exactly? How... [06:23:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:24:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:29:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:53:00] (03Abandoned) 10Nikerabbit: Localisation updates from https://translatewiki.net. [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1356827 (owner: 10L10n-bot) [06:53:25] (03Abandoned) 10Nikerabbit: Localisation updates from https://translatewiki.net. [labs/tools/wikiinfo] - 10https://gerrit.wikimedia.org/r/1356839 (owner: 10L10n-bot) [06:53:42] (03Abandoned) 10Nikerabbit: Localisation updates from https://translatewiki.net. [labs/tools/commons-mass-description] - 10https://gerrit.wikimedia.org/r/1356841 (owner: 10L10n-bot) [06:54:35] (03Abandoned) 10Nikerabbit: Localisation updates from https://translatewiki.net. [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1356844 (owner: 10L10n-bot) [06:54:40] (03Abandoned) 10Nikerabbit: Localisation updates from https://translatewiki.net. [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/1356845 (owner: 10L10n-bot) [06:54:45] (03Abandoned) 10Nikerabbit: Localisation updates from https://translatewiki.net. [labs/tools/massmailer] - 10https://gerrit.wikimedia.org/r/1356846 (owner: 10L10n-bot) [06:54:51] (03Abandoned) 10Nikerabbit: Localisation updates from https://translatewiki.net. [labs/tools/weapon-of-mass-description] - 10https://gerrit.wikimedia.org/r/1356849 (owner: 10L10n-bot) [06:54:58] (03Abandoned) 10Nikerabbit: Localisation updates from https://translatewiki.net. [labs/tools/watch-translations] - 10https://gerrit.wikimedia.org/r/1356848 (owner: 10L10n-bot) [06:55:03] (03Abandoned) 10Nikerabbit: Localisation updates from https://translatewiki.net. [labs/xtools] - 10https://gerrit.wikimedia.org/r/1356851 (owner: 10L10n-bot) [06:56:20] (03CR) 10Nikerabbit: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1356883 (owner: 10L10n-bot) [06:57:39] (03CR) 10Nikerabbit: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/wikiinfo] - 10https://gerrit.wikimedia.org/r/1356893 (owner: 10L10n-bot) [06:58:05] (03CR) 10Nikerabbit: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/commons-mass-description] - 10https://gerrit.wikimedia.org/r/1356891 (owner: 10L10n-bot) [06:59:37] (03CR) 10Nikerabbit: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1356892 (owner: 10L10n-bot) [07:00:26] (03CR) 10Nikerabbit: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/1356894 (owner: 10L10n-bot) [07:00:46] (03CR) 10Nikerabbit: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/weapon-of-mass-description] - 10https://gerrit.wikimedia.org/r/1356896 (owner: 10L10n-bot) [07:01:57] (03CR) 10Nikerabbit: [V:03+2] Localisation updates from https://translatewiki.net. [labs/xtools] - 10https://gerrit.wikimedia.org/r/1356898 (owner: 10L10n-bot) [07:08:43] (03CR) 10Nikerabbit: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/massmailer] - 10https://gerrit.wikimedia.org/r/1356895 (owner: 10L10n-bot) [07:09:25] (03CR) 10Nikerabbit: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/watch-translations] - 10https://gerrit.wikimedia.org/r/1356897 (owner: 10L10n-bot) [07:38:10] (03approved) 10tlepage: sshfs: follow symlinks [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/347 (https://phabricator.wikimedia.org/T432276) (owner: 10dcaro) [07:45:27] 10Cloud-VPS (Project-requests): Request creation of SignWriting VPS project - https://phabricator.wikimedia.org/T441738#12452721 (10Etrapani) >>! In T441738#12452279, @Bugreporter wrote: > This exists but is not maintained: {T435781} > > So it can be a request to adopt a VPS project instead of creating a new on... [08:04:11] 06tools-infrastructure-team, 06tools-platform-team: Investigate periodic bots disconnections - https://phabricator.wikimedia.org/T441765 (10fgiunchedi) 03NEW [08:19:32] 10Cloud-VPS (Project-requests): Request creation of SignWriting VPS project - https://phabricator.wikimedia.org/T441738#12452845 (10Aklapper) > Potential users are sign‑language contributors that want Wikipedia in their languages. The extension is designed to eventually replace the current gadget + image‑server... [08:22:45] (03CR) 10Majavah: [C:03+2] etcd: Support managing a SRV DNS record for etcd cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341953 (https://phabricator.wikimedia.org/T410721) (owner: 10Majavah) [08:25:19] (03Merged) 10jenkins-bot: etcd: Support managing a SRV DNS record for etcd cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341953 (https://phabricator.wikimedia.org/T410721) (owner: 10Majavah) [08:28:05] (03merge) 10dcaro: sshfs: follow symlinks [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/347 (https://phabricator.wikimedia.org/T432276) [08:29:09] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [lima-kilo] building locally jobs-api when a .tox env exists in the host code dir errors out - https://phabricator.wikimedia.org/T432276#12452907 (10dcaro) 05Open→03Resolved [08:34:34] 10Toolforge, 06tools-platform-team: [lima-kilo] building locally jobs-api when a .tox env exists in the host code dir errors out - https://phabricator.wikimedia.org/T432276#12452922 (10Volans) I don't think we should run tox on the same .tox directory from a host and a guest VM. They can be of totally diff... [08:39:28] 10Toolforge, 06tools-platform-team: [lima-kilo] building locally jobs-api when a .tox env exists in the host code dir errors out - https://phabricator.wikimedia.org/T432276#12452944 (10dcaro) >>! In T432276#12452922, @Volans wrote: > I don't think we should run tox on the same .tox directory from a host an... [08:45:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:50:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:10:36] 10Data-Services, 06tools-infrastructure-team, 06tools-platform-team: clouddb memory alert - https://phabricator.wikimedia.org/T438200#12453091 (10fnegri) > For 1022 (with glibc) if the hypothesis is right, the gap (rss - Memory_used) climbs steadily while Opened_tables rises and for 1023 (jmalloc) it should... [09:10:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:15:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:22:03] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'P{O:wmcs::openstack::eqiad1::control}' [09:24:22] FIRING: [15x] HAProxyBackendUnavailable: HAProxy service cinder-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [09:24:55] PROBLEM - SSH on cloudcontrol1006 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [09:26:10] FIRING: [2x] GaleraClusterSizeMismatch: Galera in eqiad1 has 2 nodes - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/GaleraClusterSizeMismatch - https://grafana.wikimedia.org/d/galera-cluster-summary/wmcs-openstack-eqiad-galera-cluster-summary - https://alerts.wikimedia.org/?q=alertname%3DGaleraClusterSizeMismatch [09:26:46] RECOVERY - SSH on cloudcontrol1006 is OK: SSH OK - OpenSSH_10.0p2 Debian-7+deb13u4 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [09:27:17] FIRING: [2x] JobUnavailable: Reduced availability for job mysql-galera in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [09:29:22] FIRING: [30x] HAProxyBackendUnavailable: HAProxy service cinder-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [09:29:28] PROBLEM - Host cloudcontrol1007 is DOWN: PING CRITICAL - Packet loss = 100% [09:30:49] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 07Sustainability (Incident Followup): Add monitoring for backy2 openstack backups - https://phabricator.wikimedia.org/T428893#12453215 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi This is done: alerts, dashboard and runbooks in p... [09:31:06] RECOVERY - Host cloudcontrol1007 is UP: PING OK - Packet loss = 0%, RTA = 0.34 ms [09:31:10] FIRING: [2x] GaleraClusterSizeMismatch: Galera in eqiad1 has 2 nodes - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/GaleraClusterSizeMismatch - https://grafana.wikimedia.org/d/galera-cluster-summary/wmcs-openstack-eqiad-galera-cluster-summary - https://alerts.wikimedia.org/?q=alertname%3DGaleraClusterSizeMismatch [09:32:17] FIRING: [4x] JobUnavailable: Reduced availability for job jmx_zookeeper in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [09:33:14] PROBLEM - SSH on cloudcontrol1011 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [09:34:22] FIRING: [45x] HAProxyBackendUnavailable: HAProxy service cinder-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [09:35:16] PROBLEM - Host cloudcontrol1011 is DOWN: PING CRITICAL - Packet loss = 100% [09:35:48] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'P{O:wmcs::openstack::eqiad1::control}' [09:36:04] RECOVERY - SSH on cloudcontrol1011 is OK: SSH OK - OpenSSH_10.0p2 Debian-7+deb13u4 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [09:36:06] RECOVERY - Host cloudcontrol1011 is UP: PING OK - Packet loss = 0%, RTA = 0.28 ms [09:36:10] FIRING: [3x] GaleraClusterSizeMismatch: Galera in eqiad1 has 2 nodes - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/GaleraClusterSizeMismatch - https://grafana.wikimedia.org/d/galera-cluster-summary/wmcs-openstack-eqiad-galera-cluster-summary - https://alerts.wikimedia.org/?q=alertname%3DGaleraClusterSizeMismatch [09:37:17] RESOLVED: [4x] JobUnavailable: Reduced availability for job jmx_zookeeper in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [09:39:22] FIRING: [45x] HAProxyBackendUnavailable: HAProxy service cinder-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [09:41:10] RESOLVED: [3x] GaleraClusterSizeMismatch: Galera in eqiad1 has 2 nodes - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/GaleraClusterSizeMismatch - https://grafana.wikimedia.org/d/galera-cluster-summary/wmcs-openstack-eqiad-galera-cluster-summary - https://alerts.wikimedia.org/?q=alertname%3DGaleraClusterSizeMismatch [09:44:22] RESOLVED: [45x] HAProxyBackendUnavailable: HAProxy service cinder-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [10:15:16] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Stand up an etcd cluster for requestctl usage - https://phabricator.wikimedia.org/T410721#12453456 (10taavi) 05Open→03Resolved [10:35:05] 10Cloud-VPS, 06tools-infrastructure-team: Copy requestctl known-fingerprints file to WMCS instance - https://phabricator.wikimedia.org/T441785 (10taavi) 03NEW [10:38:32] 10Data-Services, 06tools-infrastructure-team, 06tools-platform-team: clouddb memory alert - https://phabricator.wikimedia.org/T438200#12453536 (10Marostegui) Yeah 1022 s3 on glibc has a gap of +45 GiB. 1023 s3 on jemalloc is still at +2 GiB. Let me see how I can change puppet on these hosts to make this a pe... [10:59:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:04:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:19:49] 06tools-infrastructure-team: wmcs.openstack.cloudcontrol.reboot_node does not issue all required silences - https://phabricator.wikimedia.org/T441792 (10fgiunchedi) 03NEW [11:20:54] 10Cloud-VPS, 06tools-infrastructure-team: wmcs.openstack.cloudcontrol.reboot_node does not issue all required silences - https://phabricator.wikimedia.org/T441792#12453697 (10taavi) [11:21:29] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudlb.safe_reboot on hosts matched by 'A:cloudlb AND A:eqiad' [11:23:17] FIRING: JobUnavailable: Reduced availability for job cloudlb-haproxy in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [11:25:05] FIRING: [2x] HostBGPDown: BGP session for cloudlb1001 (172.20.1.2) is down - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DHostBGPDown [11:27:00] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudlb.safe_reboot (exit_code=0) on hosts matched by 'A:cloudlb AND A:eqiad' [11:28:17] RESOLVED: JobUnavailable: Reduced availability for job cloudlb-haproxy in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [11:29:03] 06tools-infrastructure-team, 06SRE Observability: Long-term audit log for metricsinfra alertmanager alerts - https://phabricator.wikimedia.org/T441794 (10fgiunchedi) 03NEW [11:30:05] RESOLVED: [4x] HostBGPDown: BGP session for cloudlb1001 (172.20.1.2) is down - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DHostBGPDown [11:38:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:40:25] 10Cloud-VPS, 06tools-infrastructure-team: wmcs.openstack.cloudlb.reboot_node does not issue all required silences - https://phabricator.wikimedia.org/T441796 (10fgiunchedi) 03NEW [11:41:28] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirtlocal1001} AND NOT P{F:kernelversion = 6.6.6}' [11:41:58] !log filippo@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D{cloudvirtlocal1001} AND NOT P{F:kernelversion = 6.6.6}' [11:43:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:46:36] (03PS1) 10Filippo Giunchedi: Exclude kernel only when requested [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1357251 [11:51:48] (03open) 10raymond-ndibe: create script and docs for migrating webservices to jobs-api [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/125 (https://phabricator.wikimedia.org/T431421) [12:01:00] (03update) 10raymond-ndibe: create script and docs for migrating webservices to jobs-api [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/125 (https://phabricator.wikimedia.org/T431421) [12:27:20] PROBLEM - Host cloudvirt1071 is DOWN: PING CRITICAL - Packet loss = 100% [12:41:39] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudweb.safe_reboot on hosts matched by 'P{O:wmcs::openstack::eqiad1::cloudweb}' [12:46:35] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudweb.safe_reboot (exit_code=0) on hosts matched by 'P{O:wmcs::openstack::eqiad1::cloudweb}' [12:53:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:57:21] (03open) 10psubhashish: remove logo from home and about [toolforge-repos/subtitler] - 10https://gitlab.wikimedia.org/toolforge-repos/subtitler/-/merge_requests/10 [12:58:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:15:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:20:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:32:56] (03open) 10psubhashish: Tome style header [toolforge-repos/subtitler] - 10https://gitlab.wikimedia.org/toolforge-repos/subtitler/-/merge_requests/11 [13:43:08] (03open) 10psubhashish: Tome style footer [toolforge-repos/subtitler] - 10https://gitlab.wikimedia.org/toolforge-repos/subtitler/-/merge_requests/12 [13:46:17] FIRING: JobUnavailable: Reduced availability for job blackbox_http in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [13:46:21] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for all workers [13:46:27] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.reboot for all workers [13:51:17] RESOLVED: JobUnavailable: Reduced availability for job blackbox_http in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [13:53:06] FIRING: InstanceDown: Project tools instance tools-k8s-worker-nfs-90 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [13:58:05] RESOLVED: InstanceDown: Project tools instance tools-k8s-worker-nfs-90 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [13:59:26] 06tools-infrastructure-team, 06Collaboration-Services, 10Infrastructure Security, 06Infrastructure-Foundations, and 7 others: Reboot w[cq]s servers to apply updates - https://phabricator.wikimedia.org/T441838 (10bking) 03NEW [14:00:10] 06tools-infrastructure-team, 06Collaboration-Services, 10Infrastructure Security, 06Infrastructure-Foundations, and 7 others: Reboot w[cq]s servers to apply updates - https://phabricator.wikimedia.org/T441838#12454745 (10bking) 05Open→03In progress p:05Triage→03Medium a:03bking [14:05:35] FIRING: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-89 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [14:08:16] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all workers [14:09:33] 06tools-infrastructure-team, 06Collaboration-Services, 10Infrastructure Security, 06Infrastructure-Foundations, and 6 others: Reboot w[cq]s servers to apply updates - https://phabricator.wikimedia.org/T441838#12454810 (10ssingh) [14:10:05] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-gateway-3, toolsbeta-test-k8s-control-12, toolsbeta-test-k8s-gateway-2, toolsbeta-test-k8s-control-11, toolsbeta-test-k8s-gateway-1, toolsbeta-test-k8s-control-10 [14:10:36] RESOLVED: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-89 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [14:10:42] (03CR) 10Majavah: [C:03+1] Exclude kernel only when requested [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1357251 (owner: 10Filippo Giunchedi) [14:11:18] (03CR) 10Andrew Bogott: [C:03+1] "thanks, I ran into this over the weekend too" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1357251 (owner: 10Filippo Giunchedi) [14:13:33] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-gateway-3, toolsbeta-test-k8s-control-12, toolsbeta-test-k8s-gateway-2, toolsbeta-test-k8s-control-11, toolsbeta-test-k8s-gateway-1, toolsbeta-test-k8s-control-10 [14:14:30] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-gateway-3, tools-k8s-control-9 [14:15:35] FIRING: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-88 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [14:17:14] FIRING: [8x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-gateway-3.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [14:17:44] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-gateway-3, tools-k8s-control-9 [14:18:24] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-gateway-2, tools-k8s-control-8 [14:20:36] RESOLVED: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-88 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [14:21:40] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-gateway-2, tools-k8s-control-8 [14:21:52] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-gateway-1, tools-k8s-control-7 [14:22:14] FIRING: [20x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-control-8.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [14:24:00] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-gateway-1, tools-k8s-control-7 [14:27:14] FIRING: [28x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-control-8.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [14:32:14] RESOLVED: [28x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-control-8.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [14:36:28] 10Data-Services, 06tools-infrastructure-team, 06tools-platform-team, 06DBA: clouddb memory alert - https://phabricator.wikimedia.org/T438200#12454977 (10Marostegui) 05In progress→03Resolved a:05fnegri→03Marostegui I've applied the change everywhere with: ` [13:44:07] marostegui@cumin1004:~$ sud... [15:03:01] 10Data-Services, 06tools-infrastructure-team, 06tools-platform-team, 06DBA: clouddb memory alert - https://phabricator.wikimedia.org/T438200#12455169 (10fnegri) Thank you @Marostegui <3 [15:03:54] (03open) 10psubhashish: Content guide [toolforge-repos/subtitler] - 10https://gitlab.wikimedia.org/toolforge-repos/subtitler/-/merge_requests/13 [15:31:40] (03CR) 10Filippo Giunchedi: [C:03+2] Exclude kernel only when requested [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1357251 (owner: 10Filippo Giunchedi) [15:32:06] FIRING: InstanceDown: Project tools instance tools-k8s-worker-nfs-71 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [15:37:06] RESOLVED: InstanceDown: Project tools instance tools-k8s-worker-nfs-71 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [15:43:36] FIRING: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-70 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [15:45:22] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.roll_reboot_mons (T441652) [15:48:16] PROBLEM - Host cloudcephmon1004 is DOWN: PING CRITICAL - Packet loss = 100% [15:48:35] RESOLVED: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-70 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [15:49:46] RECOVERY - Host cloudcephmon1004 is UP: PING OK - Packet loss = 0%, RTA = 0.27 ms [15:50:09] FIRING: CephSlowOps: Ceph cluster in eqiad has slow ops - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephSlowOps - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephSlowOps [15:51:29] 06tools-infrastructure-team, 06SRE Observability: Long-term audit log for metricsinfra alertmanager alerts - https://phabricator.wikimedia.org/T441794#12455409 (10herron) Sounds pretty reasonable to me! Do you foresee a need for longer term log storage on services beyond alertmanager? If that'd be useful we... [15:52:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [15:54:26] PROBLEM - Host cloudcephmon1005 is DOWN: PING CRITICAL - Packet loss = 100% [15:55:09] RESOLVED: CephSlowOps: Ceph cluster in eqiad has slow ops - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephSlowOps - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephSlowOps [15:56:24] RECOVERY - Host cloudcephmon1005 is UP: PING OK - Packet loss = 0%, RTA = 0.37 ms [15:59:02] PROBLEM - Host cloudcephmon1006 is DOWN: PING CRITICAL - Packet loss = 100% [16:00:30] RECOVERY - Host cloudcephmon1006 is UP: PING OK - Packet loss = 0%, RTA = 0.30 ms [16:01:09] FIRING: CephSlowOps: Ceph cluster in eqiad has slow ops - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephSlowOps - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephSlowOps [16:10:08] 10Tool-spacemedia, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: Update Toolforge Deploy to Heroku 24_0.21.20+ to get openjdk 27 support - https://phabricator.wikimedia.org/T439154#12455508 (10fnegri) a:03fnegri [16:14:06] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.roll_reboot_mons (exit_code=0) (T441652) [16:15:09] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.roll_reboot_osds (T441652) [16:17:52] PROBLEM - Host cloudcephosd1016 is DOWN: PING CRITICAL - Packet loss = 100% [16:18:29] (03update) 10psubhashish: Style guide link [toolforge-repos/subtitler] - 10https://gitlab.wikimedia.org/toolforge-repos/subtitler/-/merge_requests/14 [16:19:38] RECOVERY - Host cloudcephosd1016 is UP: PING OK - Packet loss = 0%, RTA = 0.37 ms [16:19:45] (03update) 10psubhashish: Style guide link [toolforge-repos/subtitler] - 10https://gitlab.wikimedia.org/toolforge-repos/subtitler/-/merge_requests/14 [16:21:52] PROBLEM - Host cloudcephosd1017 is DOWN: PING CRITICAL - Packet loss = 100% [16:23:56] RECOVERY - Host cloudcephosd1017 is UP: PING OK - Packet loss = 0%, RTA = 0.27 ms [16:25:42] PROBLEM - Host cloudcephosd1018 is DOWN: PING CRITICAL - Packet loss = 100% [16:27:40] RECOVERY - Host cloudcephosd1018 is UP: PING OK - Packet loss = 0%, RTA = 0.27 ms [16:28:39] RESOLVED: CephSlowOps: Ceph cluster in eqiad has slow ops - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephSlowOps - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephSlowOps [16:30:06] PROBLEM - Host cloudcephosd1019 is DOWN: PING CRITICAL - Packet loss = 100% [16:32:28] RECOVERY - Host cloudcephosd1019 is UP: PING OK - Packet loss = 0%, RTA = 0.27 ms [16:35:16] PROBLEM - Host cloudcephosd1020 is DOWN: PING CRITICAL - Packet loss = 100% [16:36:46] RECOVERY - Host cloudcephosd1020 is UP: PING OK - Packet loss = 0%, RTA = 0.28 ms [16:55:07] 10Tool-gitlab-content: gitlab-content.toolforge.org returning 404 responses for all(?) proxy requests - https://phabricator.wikimedia.org/T441864 (10bd808) 03NEW [17:00:10] 10Tool-gitlab-content: gitlab-content.toolforge.org returning 404 responses for all(?) proxy requests - https://phabricator.wikimedia.org/T441864#12455785 (10bd808) p:05Triage→03High Log events look like: `lang=json 2026-10-20T12:01:52+00:00 [gitlab-content-67ccddbbb6-9mq74] {"time":"2026-10-20T12:01:52.1730... [17:34:06] 10Tool-gitlab-content: gitlab-content.toolforge.org returning 404 responses for all(?) proxy requests - https://phabricator.wikimedia.org/T441864#12455960 (10taavi) @bd808 Based on https://superset.wikimedia.org/sqllab/?savedQueryId=1545 it appears that the tool here is not re-writing the `Host` header and so at... [17:44:53] 10Tool-gitlab-content: gitlab-content.toolforge.org returning 404 responses for all(?) proxy requests - https://phabricator.wikimedia.org/T441864#12456036 (10bd808) >>! In T441864#12455960, @taavi wrote: > @bd808 Based on https://superset.wikimedia.org/sqllab/?savedQueryId=1545 it appears that the tool here is n... [17:49:13] (03PS1) 10Andrew Bogott: drain_hypervisor: use optional --rebootprep flag [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1357351 [17:51:33] (03CR) 10CI reject: [V:04-1] drain_hypervisor: use optional --rebootprep flag [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1357351 (owner: 10Andrew Bogott) [17:56:14] (03PS2) 10Andrew Bogott: drain_hypervisor: use optional --rebootprep flag [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1357351 [17:58:54] (03CR) 10CI reject: [V:04-1] drain_hypervisor: use optional --rebootprep flag [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1357351 (owner: 10Andrew Bogott) [18:01:00] (03PS3) 10Andrew Bogott: drain_hypervisor: use optional --rebootprep flag [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1357351 [18:01:09] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [18:03:40] (03CR) 10CI reject: [V:04-1] drain_hypervisor: use optional --rebootprep flag [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1357351 (owner: 10Andrew Bogott) [18:04:47] FIRING: NodeDown: Node cloudcephosd1038 is down. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcephosd1038 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [18:05:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [18:09:47] RESOLVED: NodeDown: Node cloudcephosd1038 is down. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcephosd1038 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [18:17:06] 10Cloud-VPS (Project-requests): Request creation of SignWriting VPS project - https://phabricator.wikimedia.org/T441738#12456248 (10Etrapani) > That sounds like the extension should maybe get deployed in the long run on Wikipedias? That leads me to the question what that separate VPS project is good for. It doe... [18:22:23] (03PS1) 10Dzahn: frontend: add an optional banner from CODESEARCH_BANNER [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1357357 (https://phabricator.wikimedia.org/T268199) [18:28:57] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudservices.safe_reboot on hosts matched by 'P{O:wmcs::openstack::eqiad1::services}' [18:30:17] FIRING: [2x] JobUnavailable: Reduced availability for job pdns in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [18:30:46] PROBLEM - Host cloudservices1005 is DOWN: PING CRITICAL - Packet loss = 100% [18:31:30] (03update) 10psubhashish: Style guide link [toolforge-repos/subtitler] - 10https://gitlab.wikimedia.org/toolforge-repos/subtitler/-/merge_requests/14 [18:31:32] RECOVERY - Host cloudservices1005 is UP: PING OK - Packet loss = 0%, RTA = 0.31 ms [18:33:18] PROBLEM - Host cloudservices1006 is DOWN: PING CRITICAL - Packet loss = 100% [18:35:12] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudservices.safe_reboot (exit_code=0) on hosts matched by 'P{O:wmcs::openstack::eqiad1::services}' [18:35:16] RECOVERY - Host cloudservices1006 is UP: PING OK - Packet loss = 0%, RTA = 0.34 ms [18:40:17] RESOLVED: [2x] JobUnavailable: Reduced availability for job pdns in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [18:47:09] (03CR) 10Krinkle: frontend: add an optional banner from CODESEARCH_BANNER (033 comments) [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1357357 (https://phabricator.wikimedia.org/T268199) (owner: 10Dzahn) [18:55:20] (03PS2) 10Dzahn: frontend: add an optional banner from CODESEARCH_BANNER [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1357357 (https://phabricator.wikimedia.org/T268199) [18:55:31] (03CR) 10Dzahn: frontend: add an optional banner from CODESEARCH_BANNER (033 comments) [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1357357 (https://phabricator.wikimedia.org/T268199) (owner: 10Dzahn) [19:13:37] (03open) 10bd808: Rewrite Host header [toolforge-repos/gitlab-content] - 10https://gitlab.wikimedia.org/toolforge-repos/gitlab-content/-/merge_requests/14 (https://phabricator.wikimedia.org/T441864) [19:17:14] FIRING: [6x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-gateway-1.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [19:17:33] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:22:14] RESOLVED: [6x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-gateway-1.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [19:22:33] RESOLVED: [4x] ProbeDown: Service tools-k8s-haproxy-7:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:24:37] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.roll_reboot_osds (exit_code=0) (T441652) [19:26:17] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all workers [19:31:39] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [21:20:47] FIRING: NodeDown: Node cloudelastic1008 is down. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudelastic1008 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown