[00:04:56] FIRING: [2x] SystemdUnitFailed: prometheus-node-textfile-export_service_type.service on cumin2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:06:38] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-c3-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [01:11:11] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1328338 [01:11:11] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1328338 (owner: 10TrainBranchBot) [02:30:00] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [02:30:30] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1021.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [02:31:00] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [02:31:30] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [02:33:40] PROBLEM - Check unit status of httpbb_kubernetes_mw-api-int_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-api-int_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [02:39:30] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1021.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [02:40:00] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1021.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [02:41:30] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [02:42:00] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [03:14:56] FIRING: ProbeDown: Service ganeti3005:1811 has failed probes (tcp_ganeti_noded_ip4) - https://wikitech.wikimedia.org/wiki/Ganeti - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [03:19:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:33:40] RECOVERY - Check unit status of httpbb_kubernetes_mw-api-int_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-api-int_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [04:04:56] FIRING: [2x] SystemdUnitFailed: prometheus-node-textfile-export_service_type.service on cumin2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:06:38] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-c3-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [05:00:46] (03PS1) 10Anzx: arwikiquote: update wordmark [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1328315 (https://phabricator.wikimedia.org/T435505) [05:00:50] (03PS2) 10Anzx: Lift IP cap for Mapudungun editathon on 2026-08-29 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1328310 (https://phabricator.wikimedia.org/T435523) [05:02:22] (03PS3) 10Anzx: Lift IP cap for Mapudungun editathon on 2026-08-29 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1328310 (https://phabricator.wikimedia.org/T435523) [05:02:59] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, August 24 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1328310 (https://phabricator.wikimedia.org/T435523) (owner: 10Anzx) [05:04:29] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, August 24 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployca" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1328315 (https://phabricator.wikimedia.org/T435505) (owner: 10Anzx) [07:00:05] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260823T0700) [07:14:57] FIRING: ProbeDown: Service ganeti3005:1811 has failed probes (tcp_ganeti_noded_ip4) - https://wikitech.wikimedia.org/wiki/Ganeti - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [07:19:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:40:43] 06SRE, 10Pywikibot, 06Traffic, 10Wikidata, and 3 others: Pywikibot reports maxlag retry error - https://phabricator.wikimedia.org/T421642#12244162 (10Epidosis) Possibly related: https://www.wikidata.org/wiki/Wikidata_talk:Requests_for_comment/Mass-editing_policy#Lag_impact_of_large_running_batches [07:45:18] 06SRE, 10Pywikibot, 06Traffic, 10Wikidata, and 3 others: Pywikibot reports maxlag retry error on Wikidata - https://phabricator.wikimedia.org/T421642#12244165 (10Epidosis) [08:04:56] FIRING: [2x] SystemdUnitFailed: prometheus-node-textfile-export_service_type.service on cumin2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:06:38] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-c3-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [11:14:56] FIRING: ProbeDown: Service ganeti3005:1811 has failed probes (tcp_ganeti_noded_ip4) - https://wikitech.wikimedia.org/wiki/Ganeti - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [11:19:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:53:16] PROBLEM - Host wikikube-worker1034 is DOWN: PING CRITICAL - Packet loss = 33%, RTA = 2047.97 ms [11:53:30] RECOVERY - Host wikikube-worker1034 is UP: PING OK - Packet loss = 0%, RTA = 0.24 ms [11:56:20] (03PS1) 10Jelto: service::catalog: Set ipip_encapsulation for apertium in codfw [puppet] - 10https://gerrit.wikimedia.org/r/1328348 (https://phabricator.wikimedia.org/T420436) [11:56:22] (03PS1) 10Jelto: service::catalog: Set ipip_encapsulation for apertium in eqiad. [puppet] - 10https://gerrit.wikimedia.org/r/1328349 (https://phabricator.wikimedia.org/T420436) [12:05:01] FIRING: [2x] SystemdUnitFailed: prometheus-node-textfile-export_service_type.service on cumin2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:06:38] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-c3-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [12:36:39] (03PS1) 10Jelto: wikimedia.org: add etherpad-next and lower TTL [dns] - 10https://gerrit.wikimedia.org/r/1328350 (https://phabricator.wikimedia.org/T435509) [12:48:28] PROBLEM - Druid historical on an-druid1007 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.druid.cli.Main server historical https://wikitech.wikimedia.org/wiki/Analytics/Systems/Druid [13:01:28] RECOVERY - Druid historical on an-druid1007 is OK: PROCS OK: 1 process with command name java, args org.apache.druid.cli.Main server historical https://wikitech.wikimedia.org/wiki/Analytics/Systems/Druid [13:04:25] RESOLVED: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:53:53] FIRING: [2x] GanetiBGPDown: BGP session down between ganeti3005 and asw1-by27-esams - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPDown - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPDown [14:51:23] FIRING: [3x] CertAlmostExpired: gNMI TLS certificate for lsw1-c3-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [14:53:25] 06SRE, 10Castor, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: Completion of castor-save-workspace-cache stucks - https://phabricator.wikimedia.org/T435699#12244373 (10Mahveotm) [14:56:23] FIRING: [6x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [15:01:23] FIRING: [8x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [15:05:34] 06SRE, 10Castor, 06Collaboration-Services, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: Completion of castor-save-workspace-cache stucks - https://phabricator.wikimedia.org/T435699#12244415 (10Xqt) [15:11:23] FIRING: [9x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [15:14:57] FIRING: ProbeDown: Service ganeti3005:1811 has failed probes (tcp_ganeti_noded_ip4) - https://wikitech.wikimedia.org/wiki/Ganeti - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [15:17:36] 06SRE, 10Castor, 06Collaboration-Services, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: Completion of castor-save-workspace-cache stucks - https://phabricator.wikimedia.org/T435699#12244441 (10cscott) All jenkins jobs have been stuck for 22 hours at this point. Granted, it's over... [16:04:57] FIRING: [2x] SystemdUnitFailed: prometheus-node-textfile-export_service_type.service on cumin2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:01:06] !log urbanecm@deploy1003 mwscript-k8s job started: GrowthExperiments:fixLinkRecommendationData.php --wiki=enwiki --db-table --search-index --force --verbose # T434319 [17:01:12] T434319: Revalidate Add Link suggestions on enwiki - https://phabricator.wikimedia.org/T434319 [17:09:11] 06SRE, 10Castor, 06Collaboration-Services, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: Completion of castor-save-workspace-cache stucks - https://phabricator.wikimedia.org/T435699#12244549 (10Reedy) [17:26:55] !log urbanecm@deploy1003 mwscript-k8s job started: GrowthExperiments:fixLinkRecommendationData.php --wiki=enwiki --db-table --force --verbose --verbose # T434319 [17:27:00] T434319: Revalidate Add Link suggestions on enwiki - https://phabricator.wikimedia.org/T434319 [17:36:38] PROBLEM - Check unit status of httpbb_kubernetes_mw-web_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-web_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [17:45:57] (03PS1) 10Zabe: Set RemoteVirtualDomainsMapping for test-commons [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1328365 [17:54:57] FIRING: [2x] GanetiBGPDown: BGP session down between ganeti3005 and asw1-by27-esams - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPDown - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPDown [18:36:38] RECOVERY - Check unit status of httpbb_kubernetes_mw-web_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-web_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [18:56:19] 10ops-codfw, 06DC-Ops, 10Prod-Kubernetes, 06ServiceOps, 10ServiceOps-Upgrades-Hardware: move wikikube-worker servers in same rack - https://phabricator.wikimedia.org/T431585#12244705 (10Jhancock.wm) a:03Jhancock.wm [19:03:36] 10ops-codfw, 06SRE, 06DC-Ops, 06Infrastructure-Foundations: Q1:rack/setup/install ganeti205[1-6] - https://phabricator.wikimedia.org/T434698#12244711 (10Jhancock.wm) preseed has these servers in it, but site.pp doesn't. [19:07:22] 10ops-codfw, 06SRE, 06DC-Ops, 06Infrastructure-Foundations: Q1:rack/setup/install (2) new puppetservers - https://phabricator.wikimedia.org/T432998#12244726 (10Jhancock.wm) these servers aren't in preseed or site.pp. but there is a puppetserver in preseed only. typo? [19:10:54] 10ops-codfw, 06SRE, 06DC-Ops: Q4:rack/setup/install sretest2010 Config J 1P test host - https://phabricator.wikimedia.org/T394357#12244728 (10Jhancock.wm) 05Open→03Resolved [19:11:38] FIRING: [9x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [19:12:21] 10ops-codfw, 06SRE, 10Data-Persistence-Backup, 10database-backups, and 3 others: db2201 memory failure (was: both mysql instances crashed in the last few days) - https://phabricator.wikimedia.org/T434532#12244730 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm [19:14:57] FIRING: ProbeDown: Service ganeti3005:1811 has failed probes (tcp_ganeti_noded_ip4) - https://wikitech.wikimedia.org/wiki/Ganeti - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:15:33] 10ops-codfw, 06SRE, 06DBA, 06DC-Ops, and 2 others: db2207 down - https://phabricator.wikimedia.org/T435271#12244734 (10Jhancock.wm) emailed you a link [20:04:57] FIRING: [2x] SystemdUnitFailed: prometheus-node-textfile-export_service_type.service on cumin2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:29:22] (03PS1) 10Gergő Tisza: mediawiki: Ignore autocreate errors in backfilLocalAccounts.php [puppet] - 10https://gerrit.wikimedia.org/r/1328372 (https://phabricator.wikimedia.org/T432613) [20:32:34] (03PS2) 10Gergő Tisza: mediawiki: Ignore autocreate errors in backfilLocalAccounts.php [puppet] - 10https://gerrit.wikimedia.org/r/1328372 (https://phabricator.wikimedia.org/T432613) [20:37:39] FIRING: CoreBGPDown: Core BGP session down between cr2-codfw and cr2-eqiad (208.80.154.216) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=codfw&var-device=cr2-codfw:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr2-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [20:38:45] (03PS5) 10Andrew Bogott: cloud-vps: kick networkd if network breaks [puppet] - 10https://gerrit.wikimedia.org/r/1318779 (https://phabricator.wikimedia.org/T432426) [20:38:53] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr2-codfw:et-0/1/4 (Transport: cr2-eqiad:et-1/1/5 (Lumen, 449169461)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [20:57:14] (03PS6) 10Andrew Bogott: cloud-vps: kick networkd if network breaks [puppet] - 10https://gerrit.wikimedia.org/r/1318779 (https://phabricator.wikimedia.org/T432426) [21:31:37] 06SRE, 10Castor, 06Collaboration-Services, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: Completion of castor-save-workspace-cache stucks - https://phabricator.wikimedia.org/T435699#12244833 (10Krinkle) ... clicking through shows: > integration-castor06 is offline > {F99733172 hei... [21:35:07] 06SRE, 10Castor, 06Collaboration-Services, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: Completion of castor-save-workspace-cache stucks - https://phabricator.wikimedia.org/T435699#12244835 (10Les4353) Per https://integration.wikimedia.org/ci/computer/integration%2Dcastor06/ It se... [21:36:27] 06SRE, 10Castor, 10Cloud-Services, 06cloud-services-team, and 3 others: Completion of castor-save-workspace-cache stucks - https://phabricator.wikimedia.org/T435699#12244837 (10Les4353) The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia... [21:37:14] 06SRE, 10Castor, 06cloud-services-team, 06Collaboration-Services, and 2 others: Completion of castor-save-workspace-cache stucks - https://phabricator.wikimedia.org/T435699#12244839 (10Les4353) [21:54:57] FIRING: [2x] GanetiBGPDown: BGP session down between ganeti3005 and asw1-by27-esams - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPDown - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPDown [22:42:04] 06SRE, 06Collaboration-Services, 10Stewards-Onboarding-Tool, 10Wikimedia-Mailing-lists, 13Patch-For-Review: stewards1001 / stewards2001: automatically subscribe stewards to mailman lists (was: Enable API access for Mailman3) - https://phabricator.wikimedia.org/T351202#12244926 (10Urbanecm) [22:42:25] 06SRE, 10Internet-Archive, 06Traffic: archive.org cannot reach Wikimedia - https://phabricator.wikimedia.org/T435743 (10AlexisJazz) 03NEW [22:45:36] 06SRE, 10Internet-Archive, 06Traffic: archive.org cannot reach Wikimedia - https://phabricator.wikimedia.org/T435743#12244955 (10AlexisJazz) This is probably new. Looking at https://web.archive.org/web/20260000000000*/https://en.wikipedia.org/wiki/Main_Page I see 26-60 daily snapshots from August 16 to Augus... [23:11:38] FIRING: [9x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [23:14:57] FIRING: ProbeDown: Service ganeti3005:1811 has failed probes (tcp_ganeti_noded_ip4) - https://wikitech.wikimedia.org/wiki/Ganeti - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [23:23:32] 06SRE, 10Internet-Archive, 06Traffic, 07Regression: archive.org cannot reach Wikimedia - https://phabricator.wikimedia.org/T435743#12244988 (10AlexisJazz) [23:30:05] (03Abandoned) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1328336 (owner: 10TrainBranchBot) [23:41:07] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1328383 [23:41:07] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1328383 (owner: 10TrainBranchBot) [23:46:25] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed