[00:14:38] 10ops-codfw, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T439320 (10phaultfinder) 03NEW [00:24:57] FIRING: [2x] CertAlmostExpired: Certificate for service shellbox-timeline:4012 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#shellbox-timeline:4012 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [00:54:43] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T439320#12366625 (10phaultfinder) [01:08:53] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1345306 [01:08:53] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1345306 (owner: 10TrainBranchBot) [01:17:18] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1345306 (owner: 10TrainBranchBot) [01:17:22] FIRING: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-d2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [01:45:39] (03PS4) 10JHathaway: stdlib: upgrade to v10.1.0 [puppet] - 10https://gerrit.wikimedia.org/r/1345157 (https://phabricator.wikimedia.org/T438912) [01:45:53] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1345157 (https://phabricator.wikimedia.org/T438912) (owner: 10JHathaway) [01:49:14] (03CR) 10CI reject: [V:04-1] stdlib: upgrade to v10.1.0 [puppet] - 10https://gerrit.wikimedia.org/r/1345157 (https://phabricator.wikimedia.org/T438912) (owner: 10JHathaway) [02:00:21] !log mwpresync@deploy1003 Started scap build-images: Publishing wmf/next image [02:04:41] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T439320#12366642 (10phaultfinder) [02:06:27] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:07:54] !log mwpresync@deploy1003 Finished scap build-images: Publishing wmf/next image (duration: 07m 32s) [02:11:42] FIRING: JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:13:28] FIRING: [2x] ProbeDown: Service kafka-logging1003:9093 has failed probes (tcp_kafka_broker_tls_kafka_logging1003_ip4) - https://wikitech.wikimedia.org/wiki/Kafka/Administration#Renew_TLS_certificate - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [02:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:25:24] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:29:51] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T439320#12366646 (10phaultfinder) [03:20:44] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T439320#12366664 (10phaultfinder) [04:24:57] FIRING: [2x] CertAlmostExpired: Certificate for service shellbox-timeline:4012 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#shellbox-timeline:4012 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [04:28:02] PROBLEM - Host wikikube-worker1034 is DOWN: PING CRITICAL - Packet loss = 66%, RTA = 4691.54 ms [04:28:06] RECOVERY - Host wikikube-worker1034 is UP: PING OK - Packet loss = 0%, RTA = 0.25 ms [04:44:45] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T439320#12366690 (10phaultfinder) [05:17:22] FIRING: [3x] CertAlmostExpired: gNMI TLS certificate for lsw1-d4-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [05:50:57] (03PS2) 10Physikerwelt: Avoid timeouts when formatting large disks [puppet] - 10https://gerrit.wikimedia.org/r/1238335 (https://phabricator.wikimedia.org/T439324) [05:51:12] (03PS3) 10Physikerwelt: Avoid timeouts when formatting large disks [puppet] - 10https://gerrit.wikimedia.org/r/1238335 (https://phabricator.wikimedia.org/T439324) [06:06:27] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:13:28] FIRING: [2x] ProbeDown: Service kafka-logging1003:9093 has failed probes (tcp_kafka_broker_tls_kafka_logging1003_ip4) - https://wikitech.wikimedia.org/wiki/Kafka/Administration#Renew_TLS_certificate - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [06:25:24] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:00:05] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260927T0700) [08:24:57] FIRING: [2x] CertAlmostExpired: Certificate for service shellbox-timeline:4012 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#shellbox-timeline:4012 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [08:34:47] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T439320#12366789 (10phaultfinder) [09:10:44] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T439320#12366800 (10phaultfinder) [09:17:23] FIRING: [3x] CertAlmostExpired: gNMI TLS certificate for lsw1-d4-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [10:06:27] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:13:28] FIRING: [2x] ProbeDown: Service kafka-logging1003:9093 has failed probes (tcp_kafka_broker_tls_kafka_logging1003_ip4) - https://wikitech.wikimedia.org/wiki/Kafka/Administration#Renew_TLS_certificate - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [10:25:25] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:40:39] FIRING: [2x] TransitBGPDown: Transit BGP session down between cr2-esams and Init7 (2001:1620:1000::85) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown [10:44:56] FIRING: CoreRouterInterfaceDown: Core router interface down - cr2-esams:xe-0/1/6 (Transit: Init7 (N/A)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-esams:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [10:54:56] RESOLVED: CoreRouterInterfaceDown: Core router interface down - cr2-esams:xe-0/1/6 (Transit: Init7 (N/A)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-esams:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [10:55:39] RESOLVED: [2x] TransitBGPDown: Transit BGP session down between cr2-esams and Init7 (2001:1620:1000::85) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown [11:19:51] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T439320#12366847 (10phaultfinder) [11:20:37] (03CR) 101Veertje: "Thanks for the clarification! I’m aware that `operations/software/gerrit` is an upstream mirror, and I’ve also already addressed the viewp" [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1305800 (owner: 101Veertje) [11:33:13] (03CR) 10Paladox: "https://github.com/GerritCodeReview/gerrit is a mirror of https://gerrit.googlesource.com/gerrit." [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1305800 (owner: 101Veertje) [11:44:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T439320#12366851 (10phaultfinder) [11:54:56] PROBLEM - Host wikikube-worker1068 is DOWN: PING CRITICAL - Packet loss = 50%, RTA = 3054.47 ms [11:55:12] RECOVERY - Host wikikube-worker1068 is UP: PING OK - Packet loss = 0%, RTA = 0.25 ms [11:57:11] (03PS2) 101Veertje: Fix login form: responsive labels and mobile-friendly layout [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1305800 [11:57:54] (03PS3) 101Veertje: Fix login form: responsive labels and mobile-friendly layout [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1305800 [11:58:28] 10ops-eqiad, 06SRE, 06DC-Ops: Degraded RAID on an-worker1200 - https://phabricator.wikimedia.org/T439299#12366857 (10Jclark-ctr) T437170 Returned. I do still have drive @btullis let me know if I can swap this week [12:03:59] (03CR) 101Veertje: "PRs to the GitHub mirror have been accepted 33 times, including as recently as April this year:" [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1305800 (owner: 101Veertje) [12:06:11] (03CR) 101Veertje: "arch, I see those weren't merges. Really weird they had me go through the trouble of accepting the CLA, when they could have also automate" [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1305800 (owner: 101Veertje) [12:24:57] FIRING: [2x] CertAlmostExpired: Certificate for service shellbox-timeline:4012 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#shellbox-timeline:4012 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [12:34:42] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T439320#12366869 (10phaultfinder) [12:49:44] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T439320#12366883 (10phaultfinder) [13:05:25] RESOLVED: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:17:04] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for kafka expires in 29d 23h 49m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [13:17:23] FIRING: [3x] CertAlmostExpired: gNMI TLS certificate for lsw1-d4-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [13:19:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T439320#12366900 (10phaultfinder) [13:27:23] FIRING: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [14:06:27] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:13:28] FIRING: [2x] ProbeDown: Service kafka-logging1003:9093 has failed probes (tcp_kafka_broker_tls_kafka_logging1003_ip4) - https://wikitech.wikimedia.org/wiki/Kafka/Administration#Renew_TLS_certificate - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:16:10] FIRING: [3x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:34:47] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T439320#12366945 (10phaultfinder) [15:13:43] (03PS1) 10JHathaway: WIP: upgrade lvm module [puppet] - 10https://gerrit.wikimedia.org/r/1345334 [15:13:56] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1345334 (owner: 10JHathaway) [15:17:53] (03PS2) 10JHathaway: WIP: upgrade lvm module [puppet] - 10https://gerrit.wikimedia.org/r/1345334 [15:18:01] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1345334 (owner: 10JHathaway) [15:22:23] (03PS3) 10JHathaway: WIP: upgrade lvm module [puppet] - 10https://gerrit.wikimedia.org/r/1345334 [15:22:26] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1345334 (owner: 10JHathaway) [15:23:44] (03PS4) 10JHathaway: WIP: upgrade lvm module [puppet] - 10https://gerrit.wikimedia.org/r/1345334 [15:23:52] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1345334 (owner: 10JHathaway) [15:26:07] (03PS5) 10JHathaway: WIP: upgrade lvm module [puppet] - 10https://gerrit.wikimedia.org/r/1345334 [15:26:11] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1345334 (owner: 10JHathaway) [15:44:44] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T439320#12367004 (10phaultfinder) [16:11:41] FIRING: JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:24:57] FIRING: [2x] CertAlmostExpired: Certificate for service shellbox-timeline:4012 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#shellbox-timeline:4012 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [16:49:47] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T439320#12367047 (10phaultfinder) [17:02:08] (03PS1) 10Matthias Mullie: Squashed diff to master [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345339 [17:09:43] FIRING: RipeAtlasAnchorUnreachable: ipv4 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95133212 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [17:14:43] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv4 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95133212 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [17:17:05] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for kafka expires in 29d 19h 49m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [17:19:43] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv4 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95133212 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [17:19:51] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T439320#12367054 (10phaultfinder) [17:27:23] FIRING: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [18:01:03] FIRING: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [18:06:03] RESOLVED: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [18:13:28] FIRING: [2x] ProbeDown: Service kafka-logging1003:9093 has failed probes (tcp_kafka_broker_tls_kafka_logging1003_ip4) - https://wikitech.wikimedia.org/wiki/Kafka/Administration#Renew_TLS_certificate - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:16:27] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:24:45] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T439320#12367077 (10phaultfinder) [18:44:37] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T439320#12367078 (10phaultfinder) [19:19:42] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T439320#12367088 (10phaultfinder) [19:50:03] (03PS1) 10Zabe: reconcileTables: Add sleep option [core] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345344 (https://phabricator.wikimedia.org/T438750) [20:10:16] PROBLEM - MariaDB Replica Lag: s8 on db2198 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 600.27 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:10:16] PROBLEM - MariaDB Replica Lag: s1 on db2250 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 600.33 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:10:36] PROBLEM - MariaDB Replica Lag: s1 on db1240 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 618.72 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:11:00] PROBLEM - MariaDB Replica Lag: s8 on db1285 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 644.20 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:12:35] 10SRE-swift-storage, 06Commons, 06DBA, 10media-backups, and 6 others: old file revisions missing of File:A_Warm_Shade_of_Ivory_-_Henry_Mancini_album_cover.jpg - https://phabricator.wikimedia.org/T428406#12367116 (10Zabe) This is fixed for basically all cases. For some reason there are files with multiple f... [20:24:38] 10SRE-swift-storage, 06Commons, 06DBA, 10media-backups, and 6 others: old file revisions missing of File:A_Warm_Shade_of_Ivory_-_Henry_Mancini_album_cover.jpg - https://phabricator.wikimedia.org/T428406#12367117 (10Zabe) >>! In T428406#12367116, @Zabe wrote: > This is fixed for basically all cases. For som... [20:24:57] FIRING: [2x] CertAlmostExpired: Certificate for service shellbox-timeline:4012 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#shellbox-timeline:4012 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [20:46:48] (03Abandoned) 10Zabe: Start reading from new file tables on commons [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1270238 (https://phabricator.wikimedia.org/T416548) (owner: 10Zabe) [21:09:26] PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 188099824 and 23 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [21:10:26] RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 3041504 and 1 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [21:12:16] RECOVERY - MariaDB Replica Lag: s8 on db2198 is OK: OK slave_sql_lag Replication lag: 0.31 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:13:00] RECOVERY - MariaDB Replica Lag: s8 on db1285 is OK: OK slave_sql_lag Replication lag: 0.19 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:17:05] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for kafka expires in 29d 15h 49m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [21:23:16] PROBLEM - MariaDB Replica Lag: x3 on db2200 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 616.15 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:27:23] FIRING: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [21:40:16] RECOVERY - MariaDB Replica Lag: x3 on db2200 is OK: OK slave_sql_lag Replication lag: 0.12 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:49:42] 10ops-eqiad, 06DC-Ops: Alert for device ps1-e8-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T439340 (10phaultfinder) 03NEW [21:50:40] PROBLEM - MariaDB Replica Lag: x3 on db1216 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 634.01 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [22:00:36] RECOVERY - MariaDB Replica Lag: s1 on db1240 is OK: OK slave_sql_lag Replication lag: 0.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [22:01:40] RECOVERY - MariaDB Replica Lag: x3 on db1216 is OK: OK slave_sql_lag Replication lag: 0.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [22:06:16] RECOVERY - MariaDB Replica Lag: s1 on db2250 is OK: OK slave_sql_lag Replication lag: 0.27 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [22:13:28] FIRING: [2x] ProbeDown: Service kafka-logging1003:9093 has failed probes (tcp_kafka_broker_tls_kafka_logging1003_ip4) - https://wikitech.wikimedia.org/wiki/Kafka/Administration#Renew_TLS_certificate - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [22:16:27] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:21:58] PROBLEM - MariaDB Replica Lag: s4 on db1265 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 610.24 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [22:24:46] PROBLEM - MariaDB Replica Lag: s4 on db2239 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 613.26 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [23:21:24] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:30:05] (03Abandoned) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1345304 (owner: 10TrainBranchBot) [23:30:36] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - aux-k8s-ctrl_6443: Servers aux-k8s-ctrl2002.codfw.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [23:31:36] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [23:38:49] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1345348 [23:38:49] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1345348 (owner: 10TrainBranchBot) [23:47:30] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1345348 (owner: 10TrainBranchBot) [23:59:46] RECOVERY - MariaDB Replica Lag: s4 on db2239 is OK: OK slave_sql_lag Replication lag: 0.08 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response