[00:16:47] <icinga-wm>	 PROBLEM - SSH on netmon2002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring
[00:16:47] <icinga-wm>	 PROBLEM - librenms.wikimedia.org tls expiry on netmon2002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration
[00:16:47] <icinga-wm>	 PROBLEM - librenms.wikimedia.org requires authentication on netmon2002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration
[00:18:42] <jinxer-wm>	 FIRING: JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[00:33:37] <icinga-wm>	 RECOVERY - librenms.wikimedia.org tls expiry on netmon2002 is OK: OK - Certificate librenms.wikimedia.org will expire on Mon 09 Nov 2026 02:18:41 AM GMT +0000. https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration
[00:33:37] <icinga-wm>	 RECOVERY - SSH on netmon2002 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u10 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring
[00:33:37] <icinga-wm>	 RECOVERY - librenms.wikimedia.org requires authentication on netmon2002 is OK: HTTP OK: Status line output matched HTTP/1.1 302 - 701 bytes in 0.132 second response time https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration
[00:43:42] <jinxer-wm>	 RESOLVED: JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[01:06:51] <jinxer-wm>	 FIRING: NetworkDeviceAlarmActive: Alarm active on cr1-eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#Juniper_alarm - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DNetworkDeviceAlarmActive
[01:11:03] <wikibugs>	 (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1332266
[01:11:04] <wikibugs>	 (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1332266 (owner: 10TrainBranchBot)
[01:20:13] <wikibugs>	 (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1332266 (owner: 10TrainBranchBot)
[02:00:22] <logmsgbot>	 !log mwpresync@deploy1003 Started scap build-images: Publishing wmf/next image
[02:07:57] <logmsgbot>	 !log mwpresync@deploy1003 Finished scap build-images: Publishing wmf/next image (duration: 07m 34s)
[02:11:42] <jinxer-wm>	 FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[02:16:42] <jinxer-wm>	 RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[02:35:38] <jinxer-wm>	 FIRING: [2x] CertAlmostExpired: gNMI TLS certificate for lsw1-d6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired
[03:10:23] <jinxer-wm>	 FIRING: [2x] CertAlmostExpired: gNMI TLS certificate for lsw1-d6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired
[05:06:51] <jinxer-wm>	 FIRING: NetworkDeviceAlarmActive: Alarm active on cr1-eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#Juniper_alarm - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DNetworkDeviceAlarmActive
[07:00:05] <jouncebot>	 Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260830T0700)
[07:10:38] <jinxer-wm>	 FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-d6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired
[08:14:58] <wikibugs>	 (03PS1) 10PipelineBot: citoid: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1332460
[08:14:59] <wikibugs>	 (03PS1) 10PipelineBot: citoid: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1332461
[08:20:13] <wikibugs>	 (03CR) 101Veertje: "I had already tried to patch the absence of <label> tags in [PR 53](https://github.com/GerritCodeReview/gerrit/pull/53) on GitHub. I've no" [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1305800 (owner: 101Veertje)
[09:06:51] <jinxer-wm>	 FIRING: NetworkDeviceAlarmActive: Alarm active on cr1-eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#Juniper_alarm - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DNetworkDeviceAlarmActive
[09:10:30] <wikibugs>	 (03PS5) 10Abijeet Patro: ArticleGuidance: Configure feedback links to local talk pages [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1329314 (https://phabricator.wikimedia.org/T433483)
[10:37:11] <wikibugs>	 (03PS1) 10Revi: kowiki: convert extendedconfirmed calculation to begin from first edit [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1332474 (https://phabricator.wikimedia.org/T436449)
[10:39:27] <wikibugs>	 (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, August 31 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1332474 (https://phabricator.wikimedia.org/T436449) (owner: 10Revi)
[10:40:20] <wikibugs>	 (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, August 31 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1329314 (https://phabricator.wikimedia.org/T433483) (owner: 10Abijeet Patro)
[10:47:53] <wikibugs>	 (03Abandoned) 10Mvolz: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326687 (owner: 10PipelineBot)
[10:48:27] <wikibugs>	 (03Abandoned) 10Mvolz: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326302 (owner: 10PipelineBot)
[10:48:40] <wikibugs>	 (03Abandoned) 10Mvolz: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326301 (owner: 10PipelineBot)
[10:48:57] <wikibugs>	 (03Abandoned) 10Mvolz: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307403 (owner: 10PipelineBot)
[10:49:07] <wikibugs>	 (03Abandoned) 10Mvolz: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1248140 (owner: 10PipelineBot)
[10:49:15] <wikibugs>	 (03Abandoned) 10Mvolz: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1241851 (owner: 10PipelineBot)
[10:49:26] <wikibugs>	 (03Abandoned) 10Mvolz: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1232979 (owner: 10PipelineBot)
[10:49:36] <wikibugs>	 (03Abandoned) 10Mvolz: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1229434 (owner: 10PipelineBot)
[10:49:46] <wikibugs>	 (03Abandoned) 10Mvolz: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1213629 (owner: 10PipelineBot)
[10:49:55] <wikibugs>	 (03Abandoned) 10Mvolz: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1211775 (owner: 10PipelineBot)
[10:50:06] <wikibugs>	 (03Abandoned) 10Mvolz: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1192607 (owner: 10PipelineBot)
[10:50:20] <wikibugs>	 (03Abandoned) 10Mvolz: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1189913 (owner: 10PipelineBot)
[10:50:36] <wikibugs>	 (03Abandoned) 10Mvolz: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1187929 (owner: 10PipelineBot)
[10:51:06] <wikibugs>	 (03Abandoned) 10Mvolz: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1258158 (owner: 10PipelineBot)
[11:10:38] <jinxer-wm>	 FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-d6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired
[12:48:16] <wikibugs>	 (03Abandoned) 10Mvolz: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1205968 (owner: 10PipelineBot)
[12:49:21] <wikibugs>	 (03Abandoned) 10Mvolz: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1255708 (owner: 10PipelineBot)
[12:49:32] <wikibugs>	 (03Abandoned) 10Mvolz: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1249273 (owner: 10PipelineBot)
[12:49:45] <wikibugs>	 (03Abandoned) 10Mvolz: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1244693 (owner: 10PipelineBot)
[12:50:06] <wikibugs>	 (03Abandoned) 10Mvolz: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1240666 (owner: 10PipelineBot)
[12:50:19] <wikibugs>	 (03Abandoned) 10Mvolz: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1230294 (owner: 10PipelineBot)
[12:50:30] <wikibugs>	 (03Abandoned) 10Mvolz: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1228469 (owner: 10PipelineBot)
[12:50:40] <wikibugs>	 (03Abandoned) 10Mvolz: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1227310 (owner: 10PipelineBot)
[12:50:53] <wikibugs>	 (03Abandoned) 10Mvolz: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1207305 (owner: 10PipelineBot)
[12:51:04] <wikibugs>	 (03Abandoned) 10Mvolz: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1272689 (owner: 10PipelineBot)
[12:51:20] <wikibugs>	 (03Abandoned) 10Mvolz: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1221603 (owner: 10PipelineBot)
[12:51:34] <wikibugs>	 (03Abandoned) 10Mvolz: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1275393 (owner: 10PipelineBot)
[13:06:51] <jinxer-wm>	 FIRING: NetworkDeviceAlarmActive: Alarm active on cr1-eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#Juniper_alarm - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DNetworkDeviceAlarmActive
[15:10:38] <jinxer-wm>	 FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-d6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired
[16:00:23] <jinxer-wm>	 FIRING: [2x] CertAlmostExpired: gNMI TLS certificate for lsw1-d3-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired
[16:11:42] <jinxer-wm>	 FIRING: JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[16:16:42] <jinxer-wm>	 RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[17:06:51] <jinxer-wm>	 FIRING: NetworkDeviceAlarmActive: Alarm active on cr1-eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#Juniper_alarm - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DNetworkDeviceAlarmActive
[17:23:15] <jinxer-wm>	 FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 17.18% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy
[17:28:15] <jinxer-wm>	 RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 14.91% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy
[18:50:39] <jinxer-wm>	 FIRING: CoreBGPDown: Core BGP session down between cr1-magru and cr1-eqiad (195.200.68.136) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=magru&var-device=cr1-magru:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr1-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown
[18:55:39] <jinxer-wm>	 RESOLVED: CoreBGPDown: Core BGP session down between cr1-magru and cr1-eqiad (195.200.68.136) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=magru&var-device=cr1-magru:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr1-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown
[19:30:10] <jinxer-wm>	 FIRING: BFDdown: BFD session down between cr1-magru and 195.200.68.136 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown
[19:35:10] <jinxer-wm>	 RESOLVED: BFDdown: BFD session down between cr1-magru and 195.200.68.136 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown
[19:50:10] <jinxer-wm>	 FIRING: BFDdown: BFD session down between cr1-eqiad and fe80::b6f9:5d07:c30:c93f - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown
[19:55:10] <jinxer-wm>	 RESOLVED: BFDdown: BFD session down between cr1-eqiad and fe80::b6f9:5d07:c30:c93f - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown
[20:00:38] <jinxer-wm>	 FIRING: [2x] CertAlmostExpired: gNMI TLS certificate for lsw1-d3-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired
[20:20:37] <icinga-wm>	 PROBLEM - Host wikikube-worker1138 is DOWN: PING CRITICAL - Packet loss = 75%, RTA = 7315.77 ms
[20:21:19] <icinga-wm>	 RECOVERY - Host wikikube-worker1138 is UP: PING OK - Packet loss = 0%, RTA = 0.34 ms
[20:34:46] <wikibugs>	 06SRE, 10Beta-Cluster-Infrastructure, 06Traffic: Name new CDN servers deployment-cp-(text|upload)0x instead of deployment-cache-(text|upload)0x - https://phabricator.wikimedia.org/T280393#12269129 (10Southparkfan)
[20:36:39] <wikibugs>	 06SRE, 10Beta-Cluster-Infrastructure, 06Traffic: Beta cluster haproxy does not support `warn-blocked-traffic-after` keyword - https://phabricator.wikimedia.org/T428052#12269131 (10Southparkfan)
[20:37:04] <wikibugs>	 06SRE, 10Beta-Cluster-Infrastructure, 06Traffic: Name new CDN servers deployment-cp-(text|upload)0x instead of deployment-cache-(text|upload)0x - https://phabricator.wikimedia.org/T280393#12269134 (10Southparkfan)
[21:06:51] <jinxer-wm>	 FIRING: NetworkDeviceAlarmActive: Alarm active on cr1-eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#Juniper_alarm - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DNetworkDeviceAlarmActive
[22:02:58] <wikibugs>	 (03PS1) 10Southparkfan: deployment-prep: add svc for Wikifeeds [puppet] - 10https://gerrit.wikimedia.org/r/1332499 (https://phabricator.wikimedia.org/T436462)
[22:03:15] <wikibugs>	 (03CR) 10Southparkfan: [C:04-1] "Pending merge of https://gitlab.wikimedia.org/cloudvps-repos/deployment-prep/tofu-provisioning/-/merge_requests/21" [puppet] - 10https://gerrit.wikimedia.org/r/1332499 (https://phabricator.wikimedia.org/T436462) (owner: 10Southparkfan)
[22:09:10] <jinxer-wm>	 FIRING: BFDdown: BFD session down between cr1-magru and 195.200.68.136 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown
[22:10:39] <jinxer-wm>	 FIRING: CoreBGPDown: Core BGP session down between cr1-magru and cr1-eqiad (195.200.68.136) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=magru&var-device=cr1-magru:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr1-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown
[22:14:10] <jinxer-wm>	 RESOLVED: BFDdown: BFD session down between cr1-magru and 195.200.68.136 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown
[22:15:39] <jinxer-wm>	 RESOLVED: CoreBGPDown: Core BGP session down between cr1-magru and cr1-eqiad (195.200.68.136) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=magru&var-device=cr1-magru:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr1-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown
[23:13:22] <wikibugs>	 (03PS1) 10JHathaway: Puppet 8: Replace unscoped legacy facts in profile toolforge [puppet] - 10https://gerrit.wikimedia.org/r/1332500 (https://phabricator.wikimedia.org/T435225)
[23:14:16] <wikibugs>	 (03PS1) 10JHathaway: Puppet 8: Replace unscoped legacy facts in module rancid [puppet] - 10https://gerrit.wikimedia.org/r/1332501 (https://phabricator.wikimedia.org/T435225)
[23:15:13] <wikibugs>	 (03PS1) 10JHathaway: Puppet 8: Replace unscoped legacy facts in module rsync [puppet] - 10https://gerrit.wikimedia.org/r/1332502 (https://phabricator.wikimedia.org/T435225)
[23:16:15] <wikibugs>	 (03PS1) 10JHathaway: Puppet 8: Replace unscoped legacy facts in profile puppet_compiler [puppet] - 10https://gerrit.wikimedia.org/r/1332503 (https://phabricator.wikimedia.org/T435225)
[23:16:41] <wikibugs>	 (03PS1) 10JHathaway: Puppet 8: Replace unscoped legacy facts in module acme_chief [puppet] - 10https://gerrit.wikimedia.org/r/1332504 (https://phabricator.wikimedia.org/T435225)
[23:17:36] <wikibugs>	 (03PS1) 10JHathaway: Puppet 8: Replace unscoped legacy facts in module dnsrecursor [puppet] - 10https://gerrit.wikimedia.org/r/1332505 (https://phabricator.wikimedia.org/T435225)
[23:18:55] <wikibugs>	 (03PS1) 10JHathaway: Puppet 8: Replace unscoped legacy facts in module varnish [puppet] - 10https://gerrit.wikimedia.org/r/1332506 (https://phabricator.wikimedia.org/T435225)
[23:19:41] <wikibugs>	 (03PS1) 10JHathaway: Puppet 8: Replace unscoped legacy facts in profile cache [puppet] - 10https://gerrit.wikimedia.org/r/1332507 (https://phabricator.wikimedia.org/T435225)
[23:20:08] <wikibugs>	 (03PS1) 10JHathaway: Puppet 8: Replace unscoped legacy facts in profile dns [puppet] - 10https://gerrit.wikimedia.org/r/1332508 (https://phabricator.wikimedia.org/T435225)
[23:21:58] <wikibugs>	 (03PS1) 10JHathaway: Puppet 8: Replace unscoped legacy facts in module envoyproxy [puppet] - 10https://gerrit.wikimedia.org/r/1332509 (https://phabricator.wikimedia.org/T435225)
[23:22:50] <wikibugs>	 (03PS1) 10JHathaway: Puppet 8: Replace unscoped legacy facts in module k8s [puppet] - 10https://gerrit.wikimedia.org/r/1332510 (https://phabricator.wikimedia.org/T435225)
[23:23:50] <wikibugs>	 (03PS1) 10JHathaway: Puppet 8: Replace unscoped legacy facts in module service [puppet] - 10https://gerrit.wikimedia.org/r/1332511 (https://phabricator.wikimedia.org/T435225)
[23:24:18] <wikibugs>	 (03PS1) 10JHathaway: Puppet 8: Replace unscoped legacy facts in module bigtop [puppet] - 10https://gerrit.wikimedia.org/r/1332512 (https://phabricator.wikimedia.org/T435225)
[23:25:26] <wikibugs>	 (03PS1) 10JHathaway: Puppet 8: Replace unscoped legacy facts in module elasticsearch [puppet] - 10https://gerrit.wikimedia.org/r/1332513 (https://phabricator.wikimedia.org/T435225)
[23:25:59] <wikibugs>	 (03PS1) 10JHathaway: Puppet 8: Replace unscoped legacy facts in module opensearch [puppet] - 10https://gerrit.wikimedia.org/r/1332514 (https://phabricator.wikimedia.org/T435225)
[23:26:55] <wikibugs>	 (03PS1) 10JHathaway: Puppet 8: Replace unscoped legacy facts in profile analytics [puppet] - 10https://gerrit.wikimedia.org/r/1332515 (https://phabricator.wikimedia.org/T435225)
[23:27:56] <wikibugs>	 (03PS1) 10JHathaway: Puppet 8: Replace unscoped legacy facts in module bacula [puppet] - 10https://gerrit.wikimedia.org/r/1332516 (https://phabricator.wikimedia.org/T435225)
[23:30:01] <wikibugs>	 (03PS1) 10JHathaway: Puppet 8: Replace unscoped legacy facts in module cassandra [puppet] - 10https://gerrit.wikimedia.org/r/1332517 (https://phabricator.wikimedia.org/T435225)
[23:31:08] <wikibugs>	 (03PS1) 10JHathaway: Puppet 8: Replace unscoped legacy facts in module ceph [puppet] - 10https://gerrit.wikimedia.org/r/1332518 (https://phabricator.wikimedia.org/T435225)
[23:31:54] <wikibugs>	 (03PS1) 10JHathaway: Puppet 8: Replace unscoped legacy facts in module swift [puppet] - 10https://gerrit.wikimedia.org/r/1332519 (https://phabricator.wikimedia.org/T435225)
[23:32:59] <wikibugs>	 (03PS1) 10JHathaway: Puppet 8: Replace unscoped legacy facts in profile mariadb [puppet] - 10https://gerrit.wikimedia.org/r/1332520 (https://phabricator.wikimedia.org/T435225)
[23:33:48] <wikibugs>	 (03PS1) 10JHathaway: Puppet 8: Replace unscoped legacy facts in module graphite [puppet] - 10https://gerrit.wikimedia.org/r/1332521 (https://phabricator.wikimedia.org/T435225)
[23:34:18] <wikibugs>	 (03PS1) 10JHathaway: Puppet 8: Replace unscoped legacy facts in module icinga [puppet] - 10https://gerrit.wikimedia.org/r/1332522 (https://phabricator.wikimedia.org/T435225)
[23:35:01] <wikibugs>	 (03PS1) 10JHathaway: Puppet 8: Replace unscoped legacy facts in module prometheus [puppet] - 10https://gerrit.wikimedia.org/r/1332523 (https://phabricator.wikimedia.org/T435225)
[23:41:05] <wikibugs>	 (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1332524
[23:41:05] <wikibugs>	 (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1332524 (owner: 10TrainBranchBot)
[23:47:20] <wikibugs>	 (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1332524 (owner: 10TrainBranchBot)