[00:40:54] FIRING: [2x] CoreBGPDown: Core BGP session down between cr1-magru and cr2-eqiad (195.200.68.150) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=magru&var-device=cr1-magru:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr2-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [00:41:25] FIRING: [2x] BFDdown: BFD session down between cr2-eqiad and 195.200.68.151 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [01:11:14] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1326047 [01:11:14] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1326047 (owner: 10TrainBranchBot) [01:22:46] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1326047 (owner: 10TrainBranchBot) [02:00:19] !log mwpresync@deploy1003 Started scap build-images: Publishing wmf/next image [02:01:23] !log mwpresync@deploy1003 Finished scap build-images: Publishing wmf/next image (duration: 01m 03s) [02:08:12] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:13:12] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:18:12] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:21:38] FIRING: [2x] CertAlmostExpired: gNMI TLS certificate for lsw1-d6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [02:48:12] FIRING: [2x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [03:22:26] 10ops-eqiad, 06DC-Ops: Unresponsive management for ms-fe1018.mgmt:22 - https://phabricator.wikimedia.org/T434993 (10phaultfinder) 03NEW [03:22:28] 10ops-eqiad, 06DC-Ops: Unresponsive management for wikikube-ctrl1002.mgmt:22 - https://phabricator.wikimedia.org/T434994 (10phaultfinder) 03NEW [03:22:29] 10ops-eqiad, 06DC-Ops: Unresponsive management for dse-k8s-wdqs1002.mgmt:22 - https://phabricator.wikimedia.org/T434995 (10phaultfinder) 03NEW [03:22:30] 10ops-eqiad, 06DC-Ops: Unresponsive management for vrts1003.mgmt:22 - https://phabricator.wikimedia.org/T434996 (10phaultfinder) 03NEW [03:22:32] 10ops-eqiad, 06DC-Ops: Unresponsive management for lvs1018.mgmt:22 - https://phabricator.wikimedia.org/T434999 (10phaultfinder) 03NEW [03:22:33] 10ops-eqiad, 06DC-Ops: Unresponsive management for cirrussearch1085.mgmt:22 - https://phabricator.wikimedia.org/T434997 (10phaultfinder) 03NEW [03:22:34] 10ops-eqiad, 06DC-Ops: Unresponsive management for logging-hd1002.mgmt:22 - https://phabricator.wikimedia.org/T434998 (10phaultfinder) 03NEW [03:22:36] 10ops-eqiad, 06DC-Ops: Unresponsive management for an-worker1150.mgmt:22 - https://phabricator.wikimedia.org/T435001 (10phaultfinder) 03NEW [03:22:40] 10ops-eqiad, 06DC-Ops: Unresponsive management for mc1060.mgmt:22 - https://phabricator.wikimedia.org/T435000 (10phaultfinder) 03NEW [03:22:45] 10ops-eqiad, 06DC-Ops: Unresponsive management for wdqs1024.mgmt:22 - https://phabricator.wikimedia.org/T435003 (10phaultfinder) 03NEW [03:22:48] 10ops-eqiad, 06DC-Ops: Unresponsive management for cirrussearch1086.mgmt:22 - https://phabricator.wikimedia.org/T435004 (10phaultfinder) 03NEW [03:22:53] 10ops-eqiad, 06DC-Ops: Unresponsive management for phab1005.mgmt:22 - https://phabricator.wikimedia.org/T435002 (10phaultfinder) 03NEW [03:23:26] 10ops-eqiad, 06DC-Ops: Unresponsive management for cp1107.mgmt:22 - https://phabricator.wikimedia.org/T435006 (10phaultfinder) 03NEW [03:23:27] 10ops-eqiad, 06DC-Ops: Unresponsive management for an-worker1219.mgmt:22 - https://phabricator.wikimedia.org/T435005 (10phaultfinder) 03NEW [03:23:28] 10ops-eqiad, 06DC-Ops: Unresponsive management for cp1106.mgmt:22 - https://phabricator.wikimedia.org/T435008 (10phaultfinder) 03NEW [03:23:29] 10ops-eqiad, 06DC-Ops: Unresponsive management for an-presto1017.mgmt:22 - https://phabricator.wikimedia.org/T435007 (10phaultfinder) 03NEW [03:23:30] 10ops-eqiad, 06DC-Ops: Unresponsive management for an-worker1218.mgmt:22 - https://phabricator.wikimedia.org/T435009 (10phaultfinder) 03NEW [03:23:31] 10ops-eqiad, 06DC-Ops: Unresponsive management for ganeti1044.mgmt:22 - https://phabricator.wikimedia.org/T435010 (10phaultfinder) 03NEW [03:24:26] 10ops-eqiad, 06DC-Ops: Unresponsive management for ganeti1042.mgmt:22 - https://phabricator.wikimedia.org/T435012 (10phaultfinder) 03NEW [03:24:27] 10ops-eqiad, 06DC-Ops: Unresponsive management for clouddumps1001.mgmt:22 - https://phabricator.wikimedia.org/T435011 (10phaultfinder) 03NEW [03:24:28] 10ops-eqiad, 06DC-Ops: Unresponsive management for backup1013.mgmt:22 - https://phabricator.wikimedia.org/T435014 (10phaultfinder) 03NEW [03:24:29] 10ops-eqiad, 06DC-Ops: Unresponsive management for an-presto1018.mgmt:22 - https://phabricator.wikimedia.org/T435016 (10phaultfinder) 03NEW [03:24:31] 10ops-eqiad, 06DC-Ops: Unresponsive management for ganeti1043.mgmt:22 - https://phabricator.wikimedia.org/T435013 (10phaultfinder) 03NEW [03:24:32] 10ops-eqiad, 06DC-Ops: Unresponsive management for clouddb1016.mgmt:22 - https://phabricator.wikimedia.org/T435015 (10phaultfinder) 03NEW [03:25:27] 10ops-eqiad, 06DC-Ops: Unresponsive management for an-worker1179.mgmt:22 - https://phabricator.wikimedia.org/T435017 (10phaultfinder) 03NEW [03:48:36] FIRING: [2x] ProbeDown: Service ganeti2046:1811 has failed probes (tcp_ganeti_noded_ip4) - https://wikitech.wikimedia.org/wiki/Ganeti - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [04:40:54] FIRING: [2x] CoreBGPDown: Core BGP session down between cr1-magru and cr2-eqiad (195.200.68.150) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=magru&var-device=cr1-magru:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr2-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [04:41:25] FIRING: [2x] BFDdown: BFD session down between cr2-eqiad and 195.200.68.151 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [06:21:38] FIRING: [2x] CertAlmostExpired: gNMI TLS certificate for lsw1-d6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [06:48:27] FIRING: [2x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [07:00:05] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260816T0700) [07:48:36] FIRING: [2x] ProbeDown: Service ganeti2046:1811 has failed probes (tcp_ganeti_noded_ip4) - https://wikitech.wikimedia.org/wiki/Ganeti - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [08:36:39] FIRING: CirrusSearchTitleSuggestIndexTooOld: Some search indices that power autocomplete have not been updated recently - https://wikitech.wikimedia.org/wiki/Search/OpenSearch/Administration#Alerts/Dashboards - TODO - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchTitleSuggestIndexTooOld [08:40:54] FIRING: [2x] CoreBGPDown: Core BGP session down between cr1-magru and cr2-eqiad (195.200.68.150) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=magru&var-device=cr1-magru:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr2-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [08:41:25] FIRING: [2x] BFDdown: BFD session down between cr2-eqiad and 195.200.68.151 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [10:21:38] FIRING: [2x] CertAlmostExpired: gNMI TLS certificate for lsw1-d6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [10:48:27] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [10:58:20] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - ml-staging-ctrl_6443: Servers ml-staging-ctrl2002.codfw.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [10:59:20] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [11:05:34] FIRING: DiskSpace: Disk space build2001:9100:/ 3.536% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=build2001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [11:48:36] FIRING: [2x] ProbeDown: Service ganeti2046:1811 has failed probes (tcp_ganeti_noded_ip4) - https://wikitech.wikimedia.org/wiki/Ganeti - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:36:54] FIRING: CirrusSearchTitleSuggestIndexTooOld: Some search indices that power autocomplete have not been updated recently - https://wikitech.wikimedia.org/wiki/Search/OpenSearch/Administration#Alerts/Dashboards - TODO - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchTitleSuggestIndexTooOld [12:40:54] FIRING: [2x] CoreBGPDown: Core BGP session down between cr1-magru and cr2-eqiad (195.200.68.150) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=magru&var-device=cr1-magru:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr2-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [12:41:25] FIRING: [2x] BFDdown: BFD session down between cr2-eqiad and 195.200.68.151 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [14:06:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 16.22% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [14:11:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 20.73% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [14:12:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.93% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [14:21:38] FIRING: [2x] CertAlmostExpired: gNMI TLS certificate for lsw1-d6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [14:22:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.11% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [14:48:27] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [15:05:50] FIRING: DiskSpace: Disk space build2001:9100:/ 2.382% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=build2001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [15:32:30] FIRING: Traffic bill over quota: Alert for device cr2-eqsin.wikimedia.org - Traffic bill over quota - https://alerts.wikimedia.org/?q=alertname%3DTraffic+bill+over+quota [15:48:36] FIRING: [2x] ProbeDown: Service ganeti2046:1811 has failed probes (tcp_ganeti_noded_ip4) - https://wikitech.wikimedia.org/wiki/Ganeti - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [15:52:30] RESOLVED: Traffic bill over quota: Alert for device cr2-eqsin.wikimedia.org - Traffic bill over quota - https://alerts.wikimedia.org/?q=alertname%3DTraffic+bill+over+quota [15:52:57] 06SRE, 06Growth-Team, 10GrowthExperiments, 10MW-on-K8s, and 2 others: Periodic script failure due to "Fatal error: no version entry for `Please`" - https://phabricator.wikimedia.org/T434979#12219361 (10jnuche) @Urbanecm_WMF Thanks for the fix. MR merged now [15:53:35] 06SRE, 06Growth-Team, 10GrowthExperiments, 10MW-on-K8s, and 2 others: Periodic script failure due to "Fatal error: no version entry for `Please`" - https://phabricator.wikimedia.org/T434979#12219362 (10jnuche) p:05Unbreak!→03Triage [16:06:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.31% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [16:08:12] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:11:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.48% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [16:13:12] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:36:54] FIRING: CirrusSearchTitleSuggestIndexTooOld: Some search indices that power autocomplete have not been updated recently - https://wikitech.wikimedia.org/wiki/Search/OpenSearch/Administration#Alerts/Dashboards - TODO - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchTitleSuggestIndexTooOld [16:40:54] FIRING: [2x] CoreBGPDown: Core BGP session down between cr1-magru and cr2-eqiad (195.200.68.150) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=magru&var-device=cr1-magru:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr2-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [16:41:25] FIRING: [2x] BFDdown: BFD session down between cr2-eqiad and 195.200.68.151 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [16:54:28] PROBLEM - Check unit status of httpbb_kubernetes_mw-api-ext-next_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-api-ext-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [17:32:25] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:54:28] RECOVERY - Check unit status of httpbb_kubernetes_mw-api-ext-next_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-api-ext-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [18:21:38] FIRING: [2x] CertAlmostExpired: gNMI TLS certificate for lsw1-d6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [18:36:36] (03CR) 10Aklapper: [V:03+2 C:03+2] Add two new locale files [phabricator/translations] (wmf/stable) - 10https://gerrit.wikimedia.org/r/1325535 (https://phabricator.wikimedia.org/T412651) (owner: 10Pppery) [19:05:49] FIRING: DiskSpace: Disk space build2001:9100:/ 2.378% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=build2001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [19:39:46] (03CR) 10Clare Ming: tk_constructive_edits: use foreachwikiindblist instead and ignore errors (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1325555 (https://phabricator.wikimedia.org/T431493) (owner: 10DLynch) [19:48:36] FIRING: [2x] ProbeDown: Service ganeti2046:1811 has failed probes (tcp_ganeti_noded_ip4) - https://wikitech.wikimedia.org/wiki/Ganeti - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:53:42] (03PS3) 10Clare Ming: InstrumentConstructiveEdits: anchor all runs to the nearest `interval` [extensions/WikimediaEvents] (wmf/1.47.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1326077 (https://phabricator.wikimedia.org/T431493) [19:54:07] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, August 17 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-ite" [extensions/WikimediaEvents] (wmf/1.47.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1326077 (https://phabricator.wikimedia.org/T431493) (owner: 10Clare Ming) [20:05:15] (03CR) 10Clare Ming: "scheduled for deployment" [puppet] - 10https://gerrit.wikimedia.org/r/1325555 (https://phabricator.wikimedia.org/T431493) (owner: 10DLynch) [20:10:40] FIRING: KubernetesRsyslogDown: rsyslog on wikikube-worker1147:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=wikikube-worker1147 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [20:13:12] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [20:15:40] RESOLVED: KubernetesRsyslogDown: rsyslog on wikikube-worker1147:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=wikikube-worker1147 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [20:36:54] FIRING: CirrusSearchTitleSuggestIndexTooOld: Some search indices that power autocomplete have not been updated recently - https://wikitech.wikimedia.org/wiki/Search/OpenSearch/Administration#Alerts/Dashboards - TODO - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchTitleSuggestIndexTooOld [20:40:54] FIRING: [2x] CoreBGPDown: Core BGP session down between cr1-magru and cr2-eqiad (195.200.68.150) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=magru&var-device=cr1-magru:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr2-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [20:41:25] FIRING: [2x] BFDdown: BFD session down between cr2-eqiad and 195.200.68.151 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [20:44:23] (03PS1) 10Ssingh: admin: temp SSH key removal for jynus [puppet] - 10https://gerrit.wikimedia.org/r/1326079 [20:45:57] (03CR) 10CI reject: [V:04-1] admin: temp SSH key removal for jynus [puppet] - 10https://gerrit.wikimedia.org/r/1326079 (owner: 10Ssingh) [20:46:18] (03CR) 10KOfori: [C:03+1] "Looks good to me." [puppet] - 10https://gerrit.wikimedia.org/r/1326079 (owner: 10Ssingh) [20:47:38] (03PS2) 10Ssingh: admin: temp SSH key removal for jynus [puppet] - 10https://gerrit.wikimedia.org/r/1326079 [20:49:04] (03CR) 10Ssingh: [C:03+2] admin: temp SSH key removal for jynus [puppet] - 10https://gerrit.wikimedia.org/r/1326079 (owner: 10Ssingh) [21:21:23] FIRING: [2x] CertAlmostExpired: gNMI TLS certificate for lsw1-d6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [21:28:51] 06SRE, 10DNS, 06Traffic: [Update DNS Record Request] - wikimedia.org - Add TXT verification for Mentimeter - https://phabricator.wikimedia.org/T434620#12219487 (10Aklapper) 05Open→03Resolved Resolving per last comment. [21:32:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:48:21] 06SRE, 06Infrastructure-Foundations, 10Puppet-Infrastructure: Puppet 8 readiness - https://phabricator.wikimedia.org/T366900#12219490 (10Krinkle) [23:05:50] FIRING: DiskSpace: Disk space build2001:9100:/ 2.374% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=build2001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [23:10:36] PROBLEM - Host doh7003 is DOWN: PING CRITICAL - Packet loss = 100% [23:10:36] PROBLEM - Host doh7004 is DOWN: PING CRITICAL - Packet loss = 100% [23:10:42] RECOVERY - Host doh7004 is UP: PING OK - Packet loss = 0%, RTA = 165.67 ms [23:10:42] RECOVERY - Host doh7003 is UP: PING OK - Packet loss = 0%, RTA = 167.05 ms [23:12:45] FIRING: WidespreadPuppetFailure: Puppet has failed in magru - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [23:41:18] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1326084 [23:41:18] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1326084 (owner: 10TrainBranchBot) [23:42:45] RESOLVED: WidespreadPuppetFailure: Puppet has failed in magru - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [23:48:36] FIRING: [2x] ProbeDown: Service ganeti2046:1811 has failed probes (tcp_ganeti_noded_ip4) - https://wikitech.wikimedia.org/wiki/Ganeti - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [23:54:12] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1326084 (owner: 10TrainBranchBot) [23:55:06] 10ops-eqiad, 06SRE, 10SRE-swift-storage, 06Data-Persistence, 06DC-Ops: Q4:rack/setup/install ms-be1098, ms-be1099, ms-be1100 - https://phabricator.wikimedia.org/T424895#12219520 (10VRiley-WMF) a:03VRiley-WMF [23:55:28] 10ops-eqiad, 06SRE, 10SRE-swift-storage, 06Data-Persistence, 06DC-Ops: Q4:rack/setup/install ms-be1098, ms-be1099, ms-be1100 - https://phabricator.wikimedia.org/T424895#12219522 (10VRiley-WMF)