[00:58:47] (03CR) 10Tryvix1509: [C:03+1] "You will now need to schedule this patch for deployment. See https://wikitech.wikimedia.org/wiki/Backport_windows" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1340518 (owner: 10Chik) [01:10:57] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1340551 [01:10:57] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1340551 (owner: 10TrainBranchBot) [01:17:06] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1340551 (owner: 10TrainBranchBot) [02:00:43] !log mwpresync@deploy1003 Started scap build-images: Publishing wmf/next image [02:08:12] !log mwpresync@deploy1003 Finished scap build-images: Publishing wmf/next image (duration: 07m 29s) [02:11:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:47:39] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:06:24] FIRING: SystemdUnitFailed: production-images-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:47:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:00:04] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260913T0700) [07:04:19] FIRING: CertAlmostExpired: Certificate for service contint1002:1443 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#contint1002:1443 - https://grafana.wikimedia.org/d/K1dRhGCnz/probes-tls-dashboard - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [08:00:45] FIRING: WidespreadPuppetFailure: Puppet has failed in esams - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [08:15:45] RESOLVED: WidespreadPuppetFailure: Puppet has failed in esams - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [08:18:02] 10ops-eqsin: Inbound errors on interface cr2-eqsin:xe-0/1/1 (Transit: Tata (1028127)) - https://phabricator.wikimedia.org/T437800 (10phaultfinder) 03NEW [08:39:51] FIRING: Transit, peering or transport OUT traffic above 90% capacity - cr2-eqiad:xe-3/3/3 (Peering: Equinix (3198427-A Wikimedia-DC5-IX-01, ... [08:39:51] MAC filter)) #page - https://w.wiki/Gbyf - https://grafana.wikimedia.org/d/d968a627-b6f6-47fc-9316-e058854a4945/throughput-network-device-interfaces?var-site=eqiad+prometheus%2Fops&var-device=cr2-eqiad:9804&var-interface=xe-3%2F3%2F3 - https://alerts.wikimedia.org/?q=alertname%3DTransitPeeringTransportOutSaturation [08:44:51] RESOLVED: Transit, peering or transport OUT traffic above 90% capacity - cr2-eqiad:xe-3/3/3 (Peering: Equinix (3198427-A Wikimedia-DC5-IX-01, ... [08:44:51] MAC filter)) #page - https://w.wiki/Gbyf - https://grafana.wikimedia.org/d/d968a627-b6f6-47fc-9316-e058854a4945/throughput-network-device-interfaces?var-site=eqiad+prometheus%2Fops&var-device=cr2-eqiad:9804&var-interface=xe-3%2F3%2F3 - https://alerts.wikimedia.org/?q=alertname%3DTransitPeeringTransportOutSaturation [08:57:51] FIRING: Transit, peering or transport OUT traffic above 90% capacity - cr2-eqiad:xe-3/3/3 (Peering: Equinix (3198427-A Wikimedia-DC5-IX-01, ... [08:57:51] MAC filter)) #page - https://w.wiki/Gbyf - https://grafana.wikimedia.org/d/d968a627-b6f6-47fc-9316-e058854a4945/throughput-network-device-interfaces?var-site=eqiad+prometheus%2Fops&var-device=cr2-eqiad:9804&var-interface=xe-3%2F3%2F3 - https://alerts.wikimedia.org/?q=alertname%3DTransitPeeringTransportOutSaturation [09:07:51] RESOLVED: Transit, peering or transport OUT traffic above 90% capacity - cr2-eqiad:xe-3/3/3 (Peering: Equinix (3198427-A Wikimedia-DC5-IX-01, ... [09:07:51] MAC filter)) #page - https://w.wiki/Gbyf - https://grafana.wikimedia.org/d/d968a627-b6f6-47fc-9316-e058854a4945/throughput-network-device-interfaces?var-site=eqiad+prometheus%2Fops&var-device=cr2-eqiad:9804&var-interface=xe-3%2F3%2F3 - https://alerts.wikimedia.org/?q=alertname%3DTransitPeeringTransportOutSaturation [09:35:41] (03PS2) 10Gergő Tisza: profile::mediawiki::php: Add unserialize_callback_func php.ini var [puppet] - 10https://gerrit.wikimedia.org/r/1337377 (https://phabricator.wikimedia.org/T389402) [09:36:59] (03PS3) 10Gergő Tisza: profile::mediawiki::php: Add unserialize_callback_func php.ini var [puppet] - 10https://gerrit.wikimedia.org/r/1337377 (https://phabricator.wikimedia.org/T389402) [10:06:39] FIRING: SystemdUnitFailed: production-images-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:10:41] !incidents [10:10:42] 8341 (RESOLVED) TransitPeeringTransportOutSaturation network sre (cr2-eqiad:9804 Peering: Equinix (3198427-A Wikimedia-DC5-IX-01, MAC filter) xe-3/3/3 gnmi eqiad) [10:10:42] 8340 (RESOLVED) TransitPeeringTransportOutSaturation network sre (cr2-eqiad:9804 Peering: Equinix (3198427-A Wikimedia-DC5-IX-01, MAC filter) xe-3/3/3 gnmi eqiad) [10:10:42] 8339 (RESOLVED) TransitPeeringTransportOutSaturation network sre (cr2-eqiad:9804 Peering: Equinix (3198427-A Wikimedia-DC5-IX-01, MAC filter) xe-3/3/3 gnmi eqiad) [10:28:58] 06SRE, 10WMF-General-or-Unknown: Domain "wikipedia.ar" redirects to the wrong language - https://phabricator.wikimedia.org/T437799#12314980 (10Aklapper) `wikipedia.com.ar` redirects to `es.wikipedia.org`. This should be a one-line change in the file `modules/ncredir/files/nc_redirects.dat` in the `operations/... [10:29:11] 06SRE, 10WMF-General-or-Unknown: Domain "wikipedia.ar" redirects to Arabic instead of Spanish - https://phabricator.wikimedia.org/T437799#12314982 (10Aklapper) [10:46:20] PROBLEM - Check unit status of httpbb_kubernetes_mw-web_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-web_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [10:47:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:04:34] FIRING: CertAlmostExpired: Certificate for service contint1002:1443 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#contint1002:1443 - https://grafana.wikimedia.org/d/K1dRhGCnz/probes-tls-dashboard - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [11:36:20] RECOVERY - Check unit status of httpbb_kubernetes_mw-web_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-web_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [13:02:25] RESOLVED: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:32:31] FIRING: Traffic bill over quota: Alert for device cr2-eqsin.wikimedia.org - Traffic bill over quota - https://alerts.wikimedia.org/?q=alertname%3DTraffic+bill+over+quota [13:42:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95133216 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [13:47:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95133216 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [13:52:31] RESOLVED: Traffic bill over quota: Alert for device cr2-eqsin.wikimedia.org - Traffic bill over quota - https://alerts.wikimedia.org/?q=alertname%3DTraffic+bill+over+quota [14:02:13] 06SRE, 10WMF-General-or-Unknown: Domain "wikipedia.ar" redirects to Arabic instead of Spanish - https://phabricator.wikimedia.org/T437799#12315064 (10CoderAnimeshPathak) a:03CoderAnimeshPathak Hii, I'd like to work on this task [14:06:40] FIRING: SystemdUnitFailed: production-images-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:24:51] (03PS1) 10CoderAnimeshPathak: ncredir: Fix wikipedia.ar redirect to Spanish Wikipedia [puppet] - 10https://gerrit.wikimedia.org/r/1340598 (https://phabricator.wikimedia.org/T437799) [15:00:21] 06SRE, 10WMF-General-or-Unknown, 13Patch-For-Review: Domain "wikipedia.ar" redirects to Arabic instead of Spanish - https://phabricator.wikimedia.org/T437799#12315072 (10Nemoralis) `.ar` is Argentina's TLD, correct, but it is also the language code for Arabic. If we change it to Spanish because of Argentina,... [15:01:08] 06SRE, 10WMF-General-or-Unknown, 13Patch-For-Review: Domain "wikipedia.ar" redirects to Arabic instead of Spanish - https://phabricator.wikimedia.org/T437799#12315073 (10Nemoralis) [15:04:34] FIRING: CertAlmostExpired: Certificate for service contint1002:1443 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#contint1002:1443 - https://grafana.wikimedia.org/d/K1dRhGCnz/probes-tls-dashboard - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [15:08:35] (03CR) 10Pppery: [C:03+1] "Probably my fault, oops and thanks for catching this." [puppet] - 10https://gerrit.wikimedia.org/r/1340598 (https://phabricator.wikimedia.org/T437799) (owner: 10CoderAnimeshPathak) [15:09:11] (03CR) 10CI reject: [V:04-1] ncredir: Fix wikipedia.ar redirect to Spanish Wikipedia [puppet] - 10https://gerrit.wikimedia.org/r/1340598 (https://phabricator.wikimedia.org/T437799) (owner: 10CoderAnimeshPathak) [15:10:47] (03CR) 10CoderAnimeshPathak: "i have not done that changes due to which the test fails..." [puppet] - 10https://gerrit.wikimedia.org/r/1340598 (https://phabricator.wikimedia.org/T437799) (owner: 10CoderAnimeshPathak) [15:11:13] (03PS2) 10Pppery: ncredir: Fix wikipedia.ar redirect to Spanish Wikipedia [puppet] - 10https://gerrit.wikimedia.org/r/1340598 (https://phabricator.wikimedia.org/T437799) (owner: 10CoderAnimeshPathak) [15:27:33] 06SRE, 06Traffic, 10WMF-General-or-Unknown, 13Patch-For-Review: Domain "wikipedia.ar" redirects to Arabic instead of Spanish - https://phabricator.wikimedia.org/T437799#12315088 (10Pppery) [16:11:42] FIRING: JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [17:04:27] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, September 14 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1340505 (https://phabricator.wikimedia.org/T437772) (owner: 10Bunnypranav) [17:46:01] 06SRE, 06Traffic, 10WMF-General-or-Unknown, 13Patch-For-Review: Domain "wikipedia.ar" redirects to Arabic instead of Spanish - https://phabricator.wikimedia.org/T437799#12315228 (10Aklapper) @Nemoralis: Anyone can ask for anything but the answer might be No sometimes? :D It seemed to me that non-chapter et... [18:06:40] FIRING: SystemdUnitFailed: production-images-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:12:40] 06SRE, 10Wikimedia-Mailing-lists: New WikiLatinos Mailing List - https://phabricator.wikimedia.org/T437742#12315241 (10Ladsgroup) I would personally wait for the recognition of the UG by affcom but they are all put on hold. We should wait to see what's going to happen about it. [18:45:59] 06SRE, 06Traffic, 10WMF-General-or-Unknown, 13Patch-For-Review: Domain "wikipedia.ar" redirects to Arabic instead of Spanish - https://phabricator.wikimedia.org/T437799#12315303 (10Nemoralis) `wikipedia.br` (Brazil TLD) redirects to `br.wikipedia.org` (Breton) [19:04:34] FIRING: CertAlmostExpired: Certificate for service contint1002:1443 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#contint1002:1443 - https://grafana.wikimedia.org/d/K1dRhGCnz/probes-tls-dashboard - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [19:34:22] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-d6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [19:39:22] FIRING: [2x] CertAlmostExpired: gNMI TLS certificate for lsw1-d6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [19:44:22] FIRING: [3x] CertAlmostExpired: gNMI TLS certificate for lsw1-d4-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [19:49:22] FIRING: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-d2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [22:06:40] FIRING: SystemdUnitFailed: production-images-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:04:34] FIRING: CertAlmostExpired: Certificate for service contint1002:1443 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#contint1002:1443 - https://grafana.wikimedia.org/d/K1dRhGCnz/probes-tls-dashboard - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [23:14:39] FIRING: [2x] TransitBGPDown: Transit BGP session down between cr2-esams and LibertyGlobal (2001:730:2209:1::d52e:ba09) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown [23:14:53] FIRING: CoreRouterInterfaceDown: Core router interface down - cr2-esams:xe-0/1/7 (Transit: Liberty Global (BB00088)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-esams:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [23:35:24] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:40:50] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1340688 [23:40:51] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1340688 (owner: 10TrainBranchBot) [23:46:33] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1340688 (owner: 10TrainBranchBot) [23:49:22] FIRING: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-d2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired